Skip to content

Add configurable runtime cap support to comparison drivers and solvers#17

Open
sidhsikka1 wants to merge 2 commits into
sidh-testfrom
codex/add-runtime-cap-selection-to-run_comparison.py
Open

Add configurable runtime cap support to comparison drivers and solvers#17
sidhsikka1 wants to merge 2 commits into
sidh-testfrom
codex/add-runtime-cap-selection-to-run_comparison.py

Conversation

@sidhsikka1

Copy link
Copy Markdown
Member

Motivation

  • Provide a single, easily-configurable total runtime cap for method comparisons so each solver can be limited consistently during evaluations.
  • Implement different timeout behaviors per solver: stop IPOPT/full NLP via a callback, stop spawning GA generations when time elapses, and for the decentralized island GA convert remaining migration budget into a time-limited migration phase that transitions to consensus when expired.

Description

  • Added scripts/run_comparison.py with a TOTAL_RUNTIME_CAP_S constant and wired it into run_method_comparison via a new max_runtime_s parameter.
  • Extended run_method_comparison to accept max_runtime_s and pass it to all three solvers (centralized_nlp, centralized_ga, decentralized_island_ga).
  • Plumbed max_runtime_s into og_opts.full_nlp and added a wall-clock timeout callback to stop the trust-constr solver when the runtime cap is reached (spacecraft_libraries/og_opts.py and spacecraft_libraries/solvers/centralized_nlp.py).
  • Modified centralized GA to run one generation at a time and check elapsed wall time between generations to stop early (spacecraft_libraries/solvers/centralized_ga.py).
  • Updated decentralized island GA and GraphManager.run_island_evolution to accept max_runtime_s, scale the effective migration time by num_agents (parallelism), stop migration when the cap is hit, and then proceed to the consensus phase (spacecraft_libraries/solvers/decentralized_island_ga.py and spacecraft_libraries/graph/graph_manager.py).

Testing

  • Compiled all modified modules with python -m compileall scripts/run_comparison.py spacecraft_libraries/evaluation/comparison.py spacecraft_libraries/solvers/centralized_nlp.py spacecraft_libraries/solvers/centralized_ga.py spacecraft_libraries/solvers/decentralized_island_ga.py spacecraft_libraries/graph/graph_manager.py spacecraft_libraries/og_opts.py, which succeeded.
  • Performed a quick smoke attempt calling run_method_comparison(..., max_runtime_s=0.01), which could not complete due to missing runtime dependencies in the environment (ModuleNotFoundError: No module named 'numpy').

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 521294a82e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

agent.initialise_island(pop_size)

start = time.perf_counter()
effective_limit = None if max_runtime_s is None else max_runtime_s * self.num_agents

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Stop multiplying runtime cap by number of agents

run_island_evolution measures elapsed wall-clock time, but it scales the limit as max_runtime_s * self.num_agents even though each generation is executed serially in the same process (for agent in self.agents). This lets the decentralized solver run up to N times longer than the configured cap (e.g., ~90s with 3 agents and a 30s cap), which breaks the intended per-method runtime limit and can skew comparison results.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant