Impossible to reinitialize codegen subsystem in julia

[redeemed-sis]

I work with embedded julia (I don't use julia interpreter, I use libjulia-* libraries directly) and due to certain problems (forced termination of the adopted thread (it makes with help of jl_adopt_thread() API, which executes julia code)

julia/src/threading.c

Line 436 in 7de5585

JL_DLLEXPORT jl_gcframe_t **jl_adopt_thread(void)

It became necessary to reinitialize the code generation subsystem (calling jl_teardown_codegen() and jl_init_codegen()), since julia remained in an incorrect state when the thread was terminated (locks in the code generation subsystem were not released). However, the reinitialization fails due to the "Library already loaded" error in the sys::DynamicLibrary::addPermanentLibrary (

julia/src/jitlayers.cpp

Lines 1945 to 1948 in 7de5585

	sys::DynamicLibrary libjulia_internal_dylib = sys::DynamicLibrary::addPermanentLibrary(
	jl_libjulia_internal_handle, &ErrorStr);
	if(!ErrorStr.empty())
	report_fatal_error(llvm::Twine("FATAL: unable to dlopen libjulia-internal\n") + ErrorStr);

), when trying to download libjulia-internal (obviously, because it has already been loaded before) and as a result, abortion is called. Ignoring this error completely solves this problem. The documentation for the addPermanentLibrary function says that it can be safely called many times for the same library.

My suggestion is to ignore this error on initialization step:

sys::DynamicLibrary libjulia_internal_dylib = sys::DynamicLibrary::addPermanentLibrary(
  jl_libjulia_internal_handle, &ErrorStr);
if(!ErrorStr.empty())
    if (ErrorStr != "Library already loaded")
        report_fatal_error(llvm::Twine("FATAL: unable to dlopen libjulia-internal\n") + ErrorStr);

[vtjnash]

You are not supposed to tear down the internal state if you need to use it again, since it will introduce thread safety violations. Do not do that, and the problem should resolve itself.

[redeemed-sis]

@vtjnash I work in such pattern:

static void *task_thread(void *arg) {
  UNUSED(arg);
  jl_adopt_thread();
  // Heavy julia logic jl_load(), jl_call() etc.
}

int main() {
  pthread_t tid;
  julia_init();
  pthread_create(&tid, task_thread, NULL);
  // Wait stop command from another process
  pthread_cancel(tid);
}

If stop command received when the thread evaluate julia code (especially runtime codegen and compilation), julia context of process remains absolutely invalid. The next pthread_create, jl_adopt_thread executes without any problems after thread cancellation, but jl_call completely freezes.

static void *task_thread(void *arg) {
  UNUSED(arg);
  jl_adopt_thread();
  // Heavy julia logic jl_load(), jl_call() etc.
}

int main() {
  pthread_t tid;
  julia_init();
  pthread_create(&tid, task_thread, NULL);
  // Wait stop command from another process
  pthread_cancel(tid);
  pthread_create(&tid, task_thread, NULL); // Task starts but freezes on the first jl_call()
}

To overcome it I found a way based on thread destructors pthread_key_create, pthread_setspecific:

static pthread_key_t task_key;
static void *task_thread(void *arg) {
  pthread_setspecific(task_key, arg);
  jl_adopt_thread();
  // Heavy julia logic jl_load(), jl_call() etc.
}

static void task_destructor(void *arg) {
  jl_teardown_codegen(); // It completes fine
  jl_init_codegen(); // It aborts without my patch in jitlayers.cpp
  JL_MUTEX_INIT(&jl_codegen_lock, "jl_codegen_lock");
}

int main() {
  pthread_t tid;
  void *thread_data = NULL; // Really it is something
  julia_init();
  pthread_key_create(&task_key, task_destructor);
  pthread_create(&tid, task_thread, thread_data);
  // Wait stop command from another process
  pthread_cancel(tid);
  pthread_create(&tid, task_thread, NULL); // With patch it works fine
}

After codegen reinitialization julia remains workable, but is is only possible with pull request, attached to this issue.

P.S.
Waiting the exiting julia context is not a good idea, because we executes heavy generated julia code, which can compile over several minutes.

[vtjnash]

pthread_cancel is undefined behavior when used with threads since it introduces data races in the code. It is not permitted to be used with Julia.

[vtjnash]

There is a similar function exported by julia (jl_safepoint_suspend_thread), though you do still need to be very careful when you use it not to introduce data races

[fingolfin]

pthread_cancel also causes all kinds of leakages. By coincidence it's on Hackernews right now for that, see https://news.ycombinator.com/item?id=45233713 pointing to a post why curl removed it again as it was causing leaks.

[redeemed-sis]

@vtjnash @fingolfin
I tried jl_safepoint_suspend_thread and unfortunately it is not compatible with my case (It is segfaulted by memory access permission). I found an alternative solution of my problem and it works, but i don't know how secure is it. The solution concluded in initializing julia with --handle-signals=yes and use

ptls->io_wait = 1;
jl_atomic_store_release(&ptls->signal_request, 2);
pthread_kill(ptls->system_id, SIGUSR2);

instead of pthread_cancel. signal_request = 2 with io_wait = 1 and ptls->defer_signal = 0; leads to throwing jl_interrupt_exception to the thread to which SIGUSR2 was emitted, and julia execution stops if it even in codegen/compilation state.

julia/src/signals-unix.c

Lines 633 to 646 in e0d49ac

	if (request == 2) {
	int force = jl_check_force_sigint();
	if (force || (!ptls->defer_signal && ptls->io_wait)) {
	jl_safepoint_consume_sigint();
	if (force)
	jl_safe_printf("WARNING: Force throwing a SIGINT\n");
	// Force a throw
	jl_clear_force_sigint();
	jl_jmp_buf *saferestore = jl_get_safe_restore();
	if (saferestore) // restarting jl_ or profile
	jl_longjmp_in_ctx(sig, ctx, *saferestore);
	else
	jl_throw_in_ctx(ct, jl_interrupt_exception, sig, ctx);
	}

What do you think about such solution?

[vtjnash]

If that segfaults, you might have forgotten to register signal handlers (jl_options.handle_signals = 1)? If so, that is another time-bomb in your code waiting to happen, since it is infrequent but will cause segfaults eventually.

[redeemed-sis]

I registered signal handlers, but after I started to check SIGUSR2 approach. But now, I tried jl_safepoint_suspend_thread with jl_options.handle_signals = 1 and behavior remains the same (signal SIGSEGV: invalid permissions for mapped object). Moreover I didn't find any tracks of signal handling in jl_safepoint_suspend_thread, as I see it store into ptls->safepoint PROT_NONE pointer (it cases the higher descripted error) and wait for a necessary gc_state of thread, isn't it? Maybe I didn't notice something.

[vtjnash]

Converted to a discussion for more formatting controls, but could also move to the Discourse forum.