Skip to content

Preload and share Drive map caches across worker processes - #584

Open
vcharraut wants to merge 4 commits into
3.0from
vcha/preload-map-cache
Open

Preload and share Drive map caches across worker processes#584
vcharraut wants to merge 4 commits into
3.0from
vcha/preload-map-cache

Conversation

@vcharraut

Copy link
Copy Markdown
Collaborator

What

  • Add preload_map_cache configuration for PufferDrive.
  • Load configured maps once in the parent process before workers start.
  • Share immutable road geometry, grid maps, lane graphs, and neighbor caches with workers through fork copy-on-write.
  • Add C bindings to preload and release map-cache entries.
  • Add lightweight config_only environment initialization to avoid constructing an unnecessary driver environment.
  • Correctly manage cache ownership and cleanup across parent and worker processes.
  • Use a local fork context when preloading, including when Python 3.14 defaults to forkserver.
  • Add C and Python regression coverage for inherited cache reuse and forkserver compatibility.

Why

Each multiprocessing worker previously loaded and retained its own copy of every map. Large multi-worker training jobs therefore duplicated static map data and consumed significantly more RAM.

Preloading static geometry before forking lets workers inherit the same memory pages through copy-on-write. This restores the intended memory reduction without globally changing the application’s multiprocessing start method.

Notes

  • Disabled by default with preload_map_cache: 0.
  • Requires use_map_cache: 1.
  • Non-preloaded environments retain the platform-default multiprocessing behavior.
  • Compatible with Python 3.14 and previous Python versions on platforms supporting fork.

@Victorbares Victorbares left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Nitpick for later : I think we may want to remove use_map_cache if the preload is sufficient

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants