Sockets are overly difficult to close when a network stack reset happens

[hlef]

The network_socket_close() API returns immediately with -EAGAIN if the network stack is currently resetting. This means that one needs to call the API in a loop to ensure that sockets are properly closed after a reset. See example. We shouldn't ask users to write such an ugly loop.

@davidchisnall suggests to have the loop inside of the network_socket_close() API.

There have also been discussions in the past (among others with @nwf) about finding a good way to have all threads exit the compartment in the case of a reset. This is related to this issue. I didn't give up on this and Wes had concrete ideas for the switcher. Unfortunately, this would make it hard to have this loop inside of the TCP/IP compartment.

The loop could, however, be implemented as a simple wrapper.

One could perhaps also picture exposing a futex that tells when the reboot is done to avoid the dirty loop altogether.

[davidchisnall]

It's currently quite a nice property that compartment reset is invisible to consumers: it's equivalent to a connection drop. Exposing the epoch counter as a futex would be a nice addition, but I'd rather users didn't have care about this unless they wanted to (for example, for telemetry). Just making network_socket_close block until the network-stack reset has finished if it's in progress would be simpler, I think.

[hlef]

Agree that not exposing the reset to callers is awesome.

Unfortunately, if we end up having a mechanism that kicks all threads out of the compartment, that loop couldn't be in the compartment itself. But it could be in a wrapper outside of the compartment, and the wrapper would be the intended public interface.