Skip to content

Fix cleanup_processes ValueError on normal workflow exit#11

Open
jonmarks12 wants to merge 10 commits intoMolecularAI:publicfrom
jonmarks12:fix-runtime-cleanup
Open

Fix cleanup_processes ValueError on normal workflow exit#11
jonmarks12 wants to merge 10 commits intoMolecularAI:publicfrom
jonmarks12:fix-runtime-cleanup

Conversation

@jonmarks12
Copy link

Problem

Currently both examples and my own implementations end with a ValueError due to the dual cleanup mechanism in RunPool. The class registers cleanup_processes with atexit for safety, but also explicitly calls cleanup_processes in __exit__. This causes:

  1. First cleanup: (__exit__) successfully cleans up all processes, empties the procs list
  2. Second cleanup: (atexit) attempts to clean up again with items list intact but procs list empty
  3. Result: zip(items, procs, strict=True) raises ValueError because lists have different lengths in second cleanup

Solution

This PR modifies cleanup_processes to handle the dual cleanup scenario gracefully while improving safety:

  1. Early return for empty process list, skip cleanup when no processes remain
  2. Use strict=False to allow standard cleanup in scenarios where items is longer thanprocs
  3. Orphan process detection, continues with process cleanup when more processes than items and raises RuntimeError afterwards to indicate the mismatch

Testing

Adds a test suite (test_cleanup.py) covering:

  1. Normal case: Equal length lists (current behavior maintained)
  2. Dual cleanup case: Fewer processes than items (cleans up remaining processes without error)
  3. Orphan processes: More processes than items (kills process and raises ``RuntimeError```)

Thanks for maintaining this project! Happy to make any adjustments based on feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant