Ollama Integration

[SyedAli-eaton]

Hey everyone!

New to using API keys and everything so bear with me. Does the project support Ollama?

I have been messing with this project for a bit and can't seem to integrate Ollama, specifically model "llama3.1". I have a random "dummy key" in place of the API key since litellm expects it but does not use it for Ollama. My .env file is set as the following:

LITELLM_API_KEY="dummy-key" 
BASE_URL="http://localhost:11434/"

ORCHESTRATOR_AGENT_PROVIDER="litellm"
LOCATOR_AGENT_PROVIDER="litellm"

ENABLE_SELF_HEALING=True
USE_LLM_FOR_LOCATOR_GENERATION=True

The test fails due to the listener failing, consistent with Browser and Selenium tests. The full error listed below:

==============================================================================
The Internet Login
==============================================================================
[ ERROR ] Calling method 'end_library_keyword' of listener 'SelfhealingAgents' failed: There is no current event loop in thread 'MainThread'.
Login with valid credentials                                          | FAIL |
TimeoutError: locator.fill: Timeout 5000ms exceeded.
Call log:
  - waiting for locator('id=user')
------------------------------------------------------------------------------
The Internet Login                                                    | FAIL |
1 test, 0 passed, 1 failed
==============================================================================

Any guidance or insight would be helpful since I am trying to learn. Thank you!

[samyalabheek]

@SyedAli-eaton did you get any resolution?

[SyedAli-eaton]

@samyalabheek Not yet. I think the LiteLLM Ollama is not supported with how the payload is being delivered.

[manykarim]

Hi @SyedAli-eaton , @samyalabheek ,
sorry for the delay, I need to set up for automatic email for new issues in the future.

It will work with Ollama, you need to make sure to enter the OpenAI compatible endpoint of your Ollama instance.
e.g. BASE_URL=http://127.0.0.1:11434/v1

BASE_URL=http://127.0.0.1:11434/v1
OPENAI_API_KEY=ollama
ORCHESTRATOR_AGENT_PROVIDER="openai"
ORCHESTRATOR_AGENT_MODEL="llama3.1:latest"
LOCATOR_AGENT_PROVIDER="openai"
LOCATOR_AGENT_MODEL="llama3.1:latest"

But there is somehow a big issue with a lof of Ollama models and their tool calling capabilities.
I need to think about if we can solve it from our side (by adjusting the way "how" we trigger the tool/function calling) or if it is just a problem with the Ollama setup itself.

For example I tried with several models which should be capable of tool calling (like phi4-mini:latest or gemma3:latest) but they were not able to call the provided tool for locator creation properly.

Also llama3.1:latest is unfortunately pretty bad with the tool calls.
I'll investigate deeper.
But for now, feel free to use the provided sample data and give it a try.

[SyedAli-eaton]

Hey all,

Happy new year!

Thanks for the sample code, Many. I was able to set it up and try it out, but yes i see the issue you are talking about.

The error is the following for the Ait.robot for browser:

Ait
==============================================================================
[ ERROR ] Calling method 'end_library_keyword' of listener 'SelfhealingAgents' failed: UnexpectedModelBehavior: Exceeded maximum retries (1) for output validation
Login with valid credentials                                          | FAIL |

My thoughts on the error:
Since the underlying model (llama3.1) cannot honor function-calling, the self-healing library cannot validate the response. Therefore, it retries once. Once it fails, it throws UnexpectedModelBehavior error.

I believe with the correct structure with the required output format, the Ollama models cannot satisfy the library’s function calls.

The agent receives invalid output from the LLM which fails validation which is when it retries and fails again causing the locator generation failure.

Hope this can help out a little with the diagnostics.

[gradil]

Hi all!

Adding to this issue, I'm also having problems with toolcalling. I'm testing this library on 3 tests with different levels of broken locatores, using local inference on my machine with ollama, my best results came from the following model:

OPENAI_API_KEY=ollama

ORCHESTRATOR_AGENT_PROVIDER=openai
ORCHESTRATOR_AGENT_MODEL=qwen3-vl:32b-instruct
ORCHESTRATOR_AGENT_TEMPERATURE=0.3

LOCATOR_AGENT_PROVIDER=openai
LOCATOR_AGENT_MODEL=qwen3-vl:32b-instruct
LOCATOR_AGENT_TEMPERATURE=0.3
LOCATOR_TYPE=xpath

the first 2 tests are successfully healed, however on the third (a simple id edited like the previous but bigger context) it fails related with tool calling:

2026-01-13 17:52:55 - ModelResponse:
LLM attempts: ToolCallPart(tool_name='get_healed_locators', args='{}', tool_call_id='call_hqj2696x')

Framework response:
RetryPromptPart(content="Unknown tool name: 'get_healed_locators'. Available tools: 'final_result'")

LLM fallback attempt:
ToolCallPart(tool_name='final_result', args='{}', tool_call_id='call_v78n0zjk')

Final Error:
pydantic_ai.exceptions.ModelRetry: None of the fixed locators are valid or unique.
pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for output validation

Hope to have contributed.

[manykarim]

Thanks again @SyedAli-eaton and @gradil and sorry for the delay.
I think we have found the root cause of the problem.
There is really a lot of difference in how different Provider handle FunctionCalling - even if we are talking about the same Models (like Model X on Ollama vs Model X on OpenRouter).
I have a fix prepared in a separate branch and we will most likely release it in the upcoming days.

[manykarim]

https://github.com/MarketSquare/robotframework-selfhealing-agents/tree/feature/improve_nonopenai_model_function_calls