JIT executors are not properly freed

[YuanchengJiang]

Bug report

Bug description:

PoC:

import bdb
code = 'a = 1\nb = 2\nc = a + b'
globals_dict = {}
locals_dict = {}
debugger = bdb.Bdb()
debugger.runctx(code, globals_dict, locals_dict)
for i in range(10):
    pass
with tempfile.NamedTemporaryFile() as tmpfile:
    tmpfile.seek(0)

Config: --enable-experimental-jit=yes --with-address-sanitizer

ASan:

Traceback (most recent call last):
  File "min.py", line 9, in <module>
    with tempfile.NamedTemporaryFile() as tmpfile:
         ^^^^^^^^
NameError: name 'tempfile' is not defined. Did you mean: 'compile'? Or did you forget to import 'tempfile'?

==3445760==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 600 byte(s) in 1 object(s) allocated from:
    #0 0x7ab0b95889c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5cc52547f6b7 in _PyObject_MallocWithType ../Include/internal/pycore_object_alloc.h:46
    #2 0x5cc52547f6b7 in gc_alloc ../Python/gc.c:2327
    #3 0x5cc52547f6b7 in _PyObject_GC_NewVar ../Python/gc.c:2369
    #4 0x5cc52555de54 in make_executor_from_uops ../Python/optimizer.c:1120
    #5 0x5cc52555de54 in uop_optimize ../Python/optimizer.c:1341
    #6 0x5cc52555de54 in _PyOptimizer_Optimize ../Python/optimizer.c:136
    #7 0x5cc524f27940 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:7656
    #8 0x5cc5253e9686 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5cc5253e9686 in _PyEval_Vector ../Python/ceval.c:1997
    #10 0x5cc5253e9686 in PyEval_EvalCode ../Python/ceval.c:880
    #11 0x5cc5255a6b0e in run_eval_code_obj ../Python/pythonrun.c:1365
    #12 0x5cc5255a6b0e in run_mod ../Python/pythonrun.c:1459
    #13 0x5cc5255ab7b7 in pyrun_file ../Python/pythonrun.c:1293
    #14 0x5cc5255ab7b7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #15 0x5cc5255ac2dc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #16 0x5cc525628bdc in pymain_run_file_obj ../Modules/main.c:410
    #17 0x5cc525628bdc in pymain_run_file ../Modules/main.c:429
    #18 0x5cc525628bdc in pymain_run_python ../Modules/main.c:691
    #19 0x5cc52562a4be in Py_RunMain ../Modules/main.c:772
    #20 0x5cc52562a4be in pymain_main ../Modules/main.c:802
    #21 0x5cc52562a4be in Py_BytesMain ../Modules/main.c:826
    #22 0x7ab0b91ba1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #23 0x7ab0b91ba28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #24 0x5cc524f4cf54 in _start (cpython/build/python+0x218f54) (BuildId: 3087b1f6c97d85c049f8eaa36e3ac5b15eccf317)

SUMMARY: AddressSanitizer: 600 byte(s) leaked in 1 allocation(s).

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[picnixz]

Is there a smaller reproducer? maybe it's just when we have a NameError that this happens (or when bdb is up).

@ZeroIntensity This looks like something we already had but I don't know if it's the same issue. Do you remember a similar issue?

cc @brandtbucher for the JIT

[ZeroIntensity]

I don't remember any issues similar to this.

[YuanchengJiang]

another poc:

import base64
empty_encoded = base64.a85encode(b'')
formatted_non_utc = email.utils.format_datetime(dt_non_utc)

asan:

Traceback (most recent call last):
  File "/home/fuzz/WorkSpace/flowfusion-cpython/my_leak_pocs/912/./test.py", line 35, in <module>
    formatted_non_utc = email.utils.format_datetime(dt_non_utc)
                                                    ^^^^^^^^^^
NameError: name 'dt_non_utc' is not defined

=================================================================
==1994340==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 912 byte(s) in 1 object(s) allocated from:
    #0 0x753de56d59c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5e2822c466b7 in _PyObject_MallocWithType ../Include/internal/pycore_object_alloc.h:46
    #2 0x5e2822c466b7 in gc_alloc ../Python/gc.c:2327
    #3 0x5e2822c466b7 in _PyObject_GC_NewVar ../Python/gc.c:2369
    #4 0x5e2822d24e54 in make_executor_from_uops ../Python/optimizer.c:1120
    #5 0x5e2822d24e54 in uop_optimize ../Python/optimizer.c:1341
    #6 0x5e2822d24e54 in _PyOptimizer_Optimize ../Python/optimizer.c:136
    #7 0x5e28226ee940 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:7656
    #8 0x5e2822bb0686 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5e2822bb0686 in _PyEval_Vector ../Python/ceval.c:1997
    #10 0x5e2822bb0686 in PyEval_EvalCode ../Python/ceval.c:880
    #11 0x5e2822d6db0e in run_eval_code_obj ../Python/pythonrun.c:1365
    #12 0x5e2822d6db0e in run_mod ../Python/pythonrun.c:1459
    #13 0x5e2822d727b7 in pyrun_file ../Python/pythonrun.c:1293
    #14 0x5e2822d727b7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #15 0x5e2822d732dc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #16 0x5e2822defbdc in pymain_run_file_obj ../Modules/main.c:410
    #17 0x5e2822defbdc in pymain_run_file ../Modules/main.c:429
    #18 0x5e2822defbdc in pymain_run_python ../Modules/main.c:691
    #19 0x5e2822df14be in Py_RunMain ../Modules/main.c:772
    #20 0x5e2822df14be in pymain_main ../Modules/main.c:802
    #21 0x5e2822df14be in Py_BytesMain ../Modules/main.c:826
    #22 0x753de53071c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #23 0x753de530728a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #24 0x5e2822713f54 in _start (/home/fuzz/WorkSpace/flowfusion-cpython/cpython/build/python+0x218f54) (BuildId: 3087b1f6c97d85c049f8eaa36e3ac5b15eccf317)

SUMMARY: AddressSanitizer: 912 byte(s) leaked in 1 allocation(s).

the poc is not related; however, backtrace is almost the same. shall i regard this as the same or different one?

[picnixz]

The relation lies in the NameError.

[picnixz]

another poc:

email.utils is not imported but there is no error on your side. Can you show me the entire file, as well as the command that you executed? (is it via python test.py or something else?) I cannot reproduce this on my side.

Nevermind, I managed to reproduce it if run as a script instead the REPL:

import base64
empty_encoded = base64.a85encode(b'')
empty_encded

Changing a85encode to b64encode doesn't seem to work so the JIT-ification seems to be important here. However, changing it to b85encode also produces a crash. I'll investigate.

[picnixz]

To be clear, we also have a simple leak without address sanitizer:

$ ./configure --with-pydebug -q CC=clang --enable-experimental-jit=yes
$ make -s -j12 
$ ./python -X dev -X showrefcount crash.py
Traceback (most recent call last):
  File "[...]/cpython/crash.py", line 3, in <module>
    empty_encded
NameError: name 'empty_encded' is not defined. Did you mean: 'empty_encoded'?
[2 refs, 1 blocks]

So we're indeed having a leak somewhere. I think the executors are not properly cleaned so I'll investigate. The contents of crash.py are:

import base64
empty_encoded = base64.b85encode(b'')
empty_encded

The JIT traceback:

Optimizing b85encode ([...]/cpython/Lib/base64.py:442) at byte offset 144
   1 ADD_TO_TRACE: _START_EXECUTOR (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   2 ADD_TO_TRACE: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
72: JUMP_BACKWARD_JIT(13)
   3 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=72, operand0=0, operand1=0)
   4 ADD_TO_TRACE: _SET_IP (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   5 ADD_TO_TRACE: _CHECK_PERIODIC (0, target=72, operand0=0, operand1=0, error_target=0)
61: FOR_ITER_LIST(11)
   6 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=61, operand0=0, operand1=0)
   7 ADD_TO_TRACE: _SET_IP (0, target=61, operand0=0x7f1e704e180a, operand1=0)
   8 ADD_TO_TRACE: _ITER_CHECK_LIST (11, target=61, operand0=0, operand1=0)
   9 ADD_TO_TRACE: _GUARD_NOT_EXHAUSTED_LIST (11, target=61, operand0=0, operand1=0)
  10 ADD_TO_TRACE: _ITER_NEXT_LIST_TIER_TWO (11, target=61, operand0=0, operand1=0)
63: STORE_FAST_LOAD_FAST(3)
  11 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=63, operand0=0, operand1=0)
  12 ADD_TO_TRACE: _SET_IP (0, target=63, operand0=0x7f1e704e180e, operand1=0)
  13 ADD_TO_TRACE: _STORE_FAST (0, target=63, operand0=0, operand1=0)
  14 ADD_TO_TRACE: _LOAD_FAST (3, target=63, operand0=0, operand1=0)
64: LOAD_FAST_BORROW(0)
  15 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=64, operand0=0, operand1=0)
  16 ADD_TO_TRACE: _SET_IP (0, target=64, operand0=0x7f1e704e1810, operand1=0)
  17 ADD_TO_TRACE: _LOAD_FAST_BORROW (0, target=64, operand0=0, operand1=0)
65: BINARY_OP(0)
  18 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=65, operand0=0, operand1=0)
  19 ADD_TO_TRACE: _SET_IP (0, target=65, operand0=0x7f1e704e1812, operand1=0)
  20 ADD_TO_TRACE: _BINARY_OP (0, target=65, operand0=0, operand1=0, error_target=0)
71: LIST_APPEND(5)
  21 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=71, operand0=0, operand1=0)
  22 ADD_TO_TRACE: _SET_IP (0, target=71, operand0=0x7f1e704e181e, operand1=0)
  23 ADD_TO_TRACE: _LIST_APPEND (5, target=71, operand0=0, operand1=0, error_target=0)
  24 ADD_TO_TRACE: _JUMP_TO_TOP (0, target=0, operand0=0, operand1=0)
Created a proto-trace for b85encode ([...]/cpython/Lib/base64.py:442) at byte offset 144 -- length 24
Optimized trace (length 31):
   0 OPTIMIZED: _START_EXECUTOR (0, jump_target=20, operand0=0x556ea0b9feb0, operand1=0)
   1 OPTIMIZED: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
   2 OPTIMIZED: _SET_IP (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   3 OPTIMIZED: _CHECK_PERIODIC (0, jump_target=0, operand0=0, operand1=0, error_target=21)
   4 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=22, operand0=0, operand1=0)
   5 OPTIMIZED: _SET_IP (0, target=61, operand0=0x7f1e704e180a, operand1=0)
   6 OPTIMIZED: _ITER_CHECK_LIST (11, jump_target=23, operand0=0, operand1=0)
   7 OPTIMIZED: _GUARD_NOT_EXHAUSTED_LIST (11, jump_target=24, operand0=0, operand1=0)
   8 OPTIMIZED: _ITER_NEXT_LIST_TIER_TWO (11, jump_target=25, operand0=0, operand1=0)
   9 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=26, operand0=0, operand1=0)
  10 OPTIMIZED: _SET_IP (0, target=63, operand0=0x7f1e704e180e, operand1=0)
  11 OPTIMIZED: _STORE_FAST_0 (0, target=63, operand0=0, operand1=0)
  12 OPTIMIZED: _LOAD_FAST_3 (3, target=63, operand0=0, operand1=0)
  13 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=27, operand0=0, operand1=0)
  14 OPTIMIZED: _LOAD_FAST_BORROW_0 (0, target=64, operand0=0, operand1=0)
  15 OPTIMIZED: _SET_IP (0, target=65, operand0=0x7f1e704e1812, operand1=0)
  16 OPTIMIZED: _BINARY_OP (0, jump_target=0, operand0=0, operand1=0, error_target=28)
  17 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=29, operand0=0, operand1=0)
  18 OPTIMIZED: _LIST_APPEND (5, jump_target=0, operand0=0, operand1=0, error_target=30)
  19 OPTIMIZED: _JUMP_TO_TOP (0, jump_target=1, operand0=0, operand1=0)
  20 OPTIMIZED: _DEOPT (0, target=72, operand0=0, operand1=0)
  21 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x48, operand1=0)
  22 OPTIMIZED: _DEOPT (0, target=61, operand0=0, operand1=0)
  23 OPTIMIZED: _EXIT_TRACE (0, target=61, operand0=0x556ea0b9ff28, operand1=0)
  24 OPTIMIZED: _EXIT_TRACE (0, target=75, operand0=0x556ea0b9ff38, operand1=0)
  25 OPTIMIZED: _DEOPT (0, target=61, operand0=0, operand1=0)
  26 OPTIMIZED: _DEOPT (0, target=63, operand0=0, operand1=0)
  27 OPTIMIZED: _DEOPT (0, target=64, operand0=0, operand1=0)
  28 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x41, operand1=0)
  29 OPTIMIZED: _DEOPT (0, target=71, operand0=0, operand1=0)
  30 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x47, operand1=0)