Skip to content

fix(integration tests): Dynamic Ollama model discovery for concurrent multi-model requests#539

Merged
aiosfoundation merged 3 commits into
agiresearch:mainfrom
RyamL1221:fix/ollama-tests
Apr 13, 2026
Merged

fix(integration tests): Dynamic Ollama model discovery for concurrent multi-model requests#539
aiosfoundation merged 3 commits into
agiresearch:mainfrom
RyamL1221:fix/ollama-tests

Conversation

@RyamL1221
Copy link
Copy Markdown
Contributor

@RyamL1221 RyamL1221 commented Apr 13, 2026

fix: Dynamic Ollama model discovery for concurrent multi-model requests

Problem

When concurrent LLM requests specify different Ollama models (e.g., qwen3:1.7b and qwen3:4b), requests for models not pre-registered in config.yaml fail with a 500 error: "Selected LLMs are not all available" — even though those models are installed and available on the Ollama server.

This caused 2 of 4 concurrent Ollama CI tests to fail (test_ollama_different_models, test_ollama_some_specified_different_models).

Root Cause

LLMAdapter._initialize_llms() only registers models explicitly listed in config.yaml. The availability check (check_availability_for_selected_llm_lists) does a strict name-in-list comparison with no fallback to query the Ollama server for models that are installed but not pre-configured.

Fix

Added dynamic Ollama model discovery to LLMAdapter in aios/llm_core/adapter.py:

  • _query_ollama_available_models() — queries the Ollama server's /api/tags endpoint to get installed models
  • _dynamic_register_ollama_model() — thread-safe, idempotent method that discovers and registers an Ollama model on-demand (updates llm_configs, llms, available_llm_names, and router config)
  • Modified execute_llm_syscalls() — when a request fails the initial availability check and the unavailable models have an "ollama" backend, attempts dynamic registration before rejecting

Non-Ollama backends, already-configured models, and truly unavailable models are completely unaffected.

Files Changed

  • aios/llm_core/adapter.py — dynamic registration infrastructure + integration into execute_llm_syscalls()
  • aios/config/config.yaml.example — added qwen3:4b to the default Ollama models list (CI pulls both models)
  • requirements.txt — added hypothesis (test dependency for property-based tests)
  • tests/modules/llm/ollama/test_dynamic_registration_pbt.py — new test file with Hypothesis-based bug condition and preservation tests (runs without Ollama server, fully mocked, compatible with both pytest and unittest runners)

Testing

  • 7 property-based tests pass locally (2 bug condition + 5 preservation)
  • Tests work with both python -m pytest and python -m unittest (CI uses the latter)
  • All 4 concurrent Ollama integration tests pass in CI (test_ollama_same_model, test_ollama_different_models, test_ollama_some_specified_same_model, test_ollama_some_specified_different_models)
  • All 4 single Ollama tests continue to pass

@aiosfoundation aiosfoundation merged commit 0b682ba into agiresearch:main Apr 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants