Add configurable runtime cap support to comparison drivers and solvers#17
Add configurable runtime cap support to comparison drivers and solvers#17sidhsikka1 wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 521294a82e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| agent.initialise_island(pop_size) | ||
|
|
||
| start = time.perf_counter() | ||
| effective_limit = None if max_runtime_s is None else max_runtime_s * self.num_agents |
There was a problem hiding this comment.
Stop multiplying runtime cap by number of agents
run_island_evolution measures elapsed wall-clock time, but it scales the limit as max_runtime_s * self.num_agents even though each generation is executed serially in the same process (for agent in self.agents). This lets the decentralized solver run up to N times longer than the configured cap (e.g., ~90s with 3 agents and a 30s cap), which breaks the intended per-method runtime limit and can skew comparison results.
Useful? React with 👍 / 👎.
Motivation
Description
scripts/run_comparison.pywith aTOTAL_RUNTIME_CAP_Sconstant and wired it intorun_method_comparisonvia a newmax_runtime_sparameter.run_method_comparisonto acceptmax_runtime_sand pass it to all three solvers (centralized_nlp,centralized_ga,decentralized_island_ga).max_runtime_sintoog_opts.full_nlpand added a wall-clock timeout callback to stop thetrust-constrsolver when the runtime cap is reached (spacecraft_libraries/og_opts.pyandspacecraft_libraries/solvers/centralized_nlp.py).spacecraft_libraries/solvers/centralized_ga.py).GraphManager.run_island_evolutionto acceptmax_runtime_s, scale the effective migration time bynum_agents(parallelism), stop migration when the cap is hit, and then proceed to the consensus phase (spacecraft_libraries/solvers/decentralized_island_ga.pyandspacecraft_libraries/graph/graph_manager.py).Testing
python -m compileall scripts/run_comparison.py spacecraft_libraries/evaluation/comparison.py spacecraft_libraries/solvers/centralized_nlp.py spacecraft_libraries/solvers/centralized_ga.py spacecraft_libraries/solvers/decentralized_island_ga.py spacecraft_libraries/graph/graph_manager.py spacecraft_libraries/og_opts.py, which succeeded.run_method_comparison(..., max_runtime_s=0.01), which could not complete due to missing runtime dependencies in the environment (ModuleNotFoundError: No module named 'numpy').Codex Task