Signal handling doesn't work in interactive Python session

[nsoblath]

Currently, to use the scarab-based signal handling you have to create a signal_handler and add the appropriate cancelable. This is fine for executables, e.g. dl-serve. However, it's not workable for an interactive session.

I created a script that uses core.Interface to e.g. get peaches from the KVS. For network reasons it wasn't able to contact the broker, so it tried multiple times using exponential backoff. I attempted to cancel that with ctrl-c, but it didn't work because the signal hadn't been setup to be caught. I got it to stop by repeating ctrl-c twice, resulting in an unhandled exception and a long traceback.

I'm not sure what the right way to handle this is, at the moment.

[nsoblath]

@laroque has pointed out this, which might be useful: https://pybind11.readthedocs.io/en/stable/faq.html#how-can-i-properly-handle-ctrl-c-in-long-running-functions

TL;DR: if the GIL is locked for a long-running function, the interpreter will hold the ctrl-c signal until the GIL is released. So the long-running function won't cancel. It has a way to get around that.

In principle the long-running py-bound C++ functions should release the GIL before they drop into the C++. We should look into whether the GIL is locked when the exponential backoff I described was active. Maybe something doesn't release the GIL that needs to.

[nsoblath]

This problem may be related to an issue in pybind11 that made it impossible to get a valid std::weak_ptr in certain situations. This is being addressed in pybind/pybind11#5624. This pybind11 issue made it impossible to cancel executables with SIGINT.

[nsoblath]

Note that this problem still exists as of 3.13.5. ctrl-c doesn't work in the exponential backup loop.