ci: crewai instrumentation tests failing on main (VCR cassette + OpenAI API connection error)

[ralf0131]

Problem

The loongsuite-instrumentation-crewai test suite is failing on the main branch. This is a pre-existing issue on main, not introduced by any specific PR.

Failing CI Runs

Run	Branch	Conclusion	Date
#28212953288	main	❌ failure	2026-06-26 02:18 UTC
#28212331441	main	❌ failure	2026-06-26 02:00 UTC

Last passing run: #28080659127 on 2026-06-24 06:49 UTC ✅

Affected Versions

All Python versions: 3.10, 3.11, 3.12, 3.13 (Ubuntu)

Failure Details

1 failed, 47 passed — the failing test is:

FAILED instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test_error_scenarios.py::TestErrorScenarios::test_api_key_missing

Error chain:

1. VCR cassette overwrite failure:

   WARNING LiteLLM: Failed to fetch remote model cost map from
   https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json:
   Can't overwrite existing cassette
   (.../tests/cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml)
   in your current record mode (<RecordMode.NONE: 'none'>).
   

2. VCR matcher failures:

   Matchers failed:
   method - assertion failure
   host - assertion failure
   path - assertion failure
   

3. OpenAI API connection error (fallback to live API):

   ERROR root:completion.py:1772 Failed to connect to OpenAI API: Connection error.
   ERROR root:completion.py:452 OpenAI API call failed: Failed to connect to OpenAI API: Connection error.
   ERROR crewai.flow.runtime:__init__.py:2978 Error executing listener call_llm_and_parse: Failed to connect to OpenAI API: Connection error.
   

Root Cause Hypothesis

The VCR cassette for test_agent_workflow.TestAgentWorkflow.test_hierarchical_workflow no longer matches the HTTP requests being made. When the cassette match fails, the test falls back to making a real OpenAI API call, which fails in CI (no network access to OpenAI API).

This likely started after one of these commits merged between 2026-06-24 and 2026-06-26:

• 1b25fcc5 feat(deepagents): support DeepAgents instrumentation (feat(deepagents): support DeepAgents instrumentation #228)
• e4232fdc feat(claude-agent-sdk): capture gen_ai.skill.* on Skill load execute_tool span (feat(claude-agent-sdk): capture gen_ai.skill.* on Skill load execute_tool span #226)

A dependency update (likely crewai or litellm) may have changed the HTTP request format, causing the cassette matchers (method/host/path) to fail.

Suggested Fix

1. Re-record the VCR cassette for test_hierarchical_workflow with the current dependency versions
2. Or update the cassette matchers to be more lenient (e.g., match on uri instead of method + host + path)
3. Or pin the crewai/litellm version to match the cassette recording

Impact

• Blocks PRs that trigger the crewai test matrix (e.g., #229)
• The test_api_key_missing test in test_error_scenarios.py is also failing as a side effect

---

Reported by github-manager-bot

[ralf0131]

Issue Evaluation

Category: bug | Status: Confirmed

The reported CI failure has been verified against the current main branch.

Verification (direct CI evidence):
The LoongSuite Test 0 workflow is failing on main:

• Run #28212953288 — ❌ failure (2026-06-26 02:18 UTC)
• Run #28212331441 — ❌ failure (2026-06-26 02:00 UTC)
• Last passing: #28080659127 (2026-06-24 06:49 UTC) ✅

Root Cause: The VCR cassette for test_agent_workflow.TestAgentWorkflow.test_hierarchical_workflow no longer matches outgoing HTTP requests (method/host/path matcher failures). When cassette matching fails, the test falls back to a live OpenAI API call, which fails in CI (no network egress to the OpenAI API). This aligns with the issue hypothesis — likely triggered by a crewai/litellm dependency change merged between 2026-06-24 and 2026-06-26 (1b25fcc5 #228, e4232fdc #226).

Impact: Blocks the LoongSuite Test 0 CI gate on main for all Python versions (3.10–3.13). No runtime impact on released packages.
Severity: medium (CI-only; blocks merge confidence but does not affect users)

Fix Assessment:
Two approaches were identified, each with a trade-off:

1. Re-record the VCR cassette against current dependency versions — requires transient OpenAI API access (an API key + network egress), which the CI/agent environment does not have.
2. Relax cassette matchers / add fail-fast so a mismatch raises a deterministic error instead of silently falling back to a live API call — a code change, but it alters test semantics and needs a maintainer decision on the desired strictness.

Because the preferred fix (cassette re-recording) depends on API access that is not available in this environment, and the alternative changes test behavior in a way that warrants a design decision, this issue is flagged for maintainer confirmation on the preferred approach before an automated PR is generated.

Next steps: A maintainer can reply with the chosen approach (e.g. /approve to proceed with the fail-fast matcher fix, or re-record the cassette manually). No associated PR exists yet.

---

Automated evaluation by github-manager-bot