Can't Reproduce AIME24 Results following readme.

[dercaft]

Thank you for your work on this exciting project!

I've been experimenting with the latest commit (9e69192) and have been carefully following the instructions in the README.md.

Following the README, the accuracy I'm seeing is quite low (1/30).

Then I test combination of different models for agents and tools:

Planner	Planner_fixed	Code	Other agents and tools	Correct tasks	Acc
agentflow-planner-7b	Qwen2.5-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	1/30	3.3%
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	1/30	3.3%
agentflow-planner-7b	Qwen2.5-7B-Instruct	Gemini-2.5-pro	Qwen2.5-7B-Instruct	2/30	6.7%
GPT-4o	Qwen2.5-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	2/30	6.7%
Gemini-2.5-pro	Qwen2.5-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	1/30	3.3%
GPT-4o	GPT-4o	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	6/30	20.0%
Gemini-2.5-pro	Gemini-2.5-pro	Qwen2.5-Coder-7B-Instruct	Qwen2.5-7B-Instruct	6/30	20.0%
Gemini-2.5-pro	Gemini-2.5-pro	Gemini-2.5-pro	Gemini-2.5-pro	17/30	56.7%

PS: Gemini-2.5-pro official performance on AIME24 is 92%.

Could you please provide any guidance on this?

Thank you for your time and any help you can offer!

[IsaacGHX]

Hi @dercaft

Thank you for your interest in AgentFlow and for conducting such thorough experiments!

Log Files and Deployment Details

To better understand your experimental setup and help diagnose the discrepancy, could you please provide your runtime log files? This would help us see:

• The exact execution flow for each problem
• Any errors or warnings during agent execution
• The specific prompts and responses from each module
• Token usage and whether any modules hit output length limits

Additionally, we strongly recommend pulling the latest version of the repository before running experiments, as there may have been updates since commit 9e69192 that could affect performance.

When sharing logs or result JSON, please include:

• Full console output during execution
• The generated results file shows predictions
• Any error messages or API rate limit warnings
• Your exact configuration files (model settings, agent configurations, etc.)

Regarding Gemini-2.5-Pro Performance

Your observation about Gemini-2.5-Pro achieving 56.7% (17/30) versus the official 92% on AIME24 is expected, and there are several important factors to consider:

We have not extensively tested Gemini-2.5-Pro within our system due to:

1. Severe rate-limiting issues - Gemini models often have restrictive quotas that cause problems during batch evaluation
2. Frequent failures to generate outputs - The model sometimes returns empty responses or errors

Key differences that explain the performance gap:

1. Quota/Rate Limit Issues: Some modules may not receive responses due to exceeding API quotas or hitting rate limits, leading to incomplete reasoning chains

2. Output Length Constraints: Our system imposes output length limits on each module. We do not enable any "thinking with <think></think>" or extended reasoning mode, meaning:

   • The model cannot use unlimited output tokens for deep reasoning
   • Responses are constrained to much shorter lengths than in the official benchmark
   • The official 92% likely uses extended thinking with much longer inference chains

3. No Extended Thinking Mode: Without the thinking/reasoning mode enabled, the model cannot perform the kind of extended chain-of-thought reasoning that achieves potential 92% performance in the official report.

Recommendation: If you'd like to verify Gemini-2.5-Pro's standalone performance on AIME24, we suggest:

• Testing it directly with full thinking mode enabled
• Using unlimited output tokens
• Running without the multi-agent framework constraints as a baseline

This would help clarify whether the 92% → 56.7% gap is due to framework constraints, quota issues, or integration challenges.