This documents a working local AI inference stack built on Orange Pi 5 Max (OP5M) single board computers, running a full web-search-capable LLM pipeline entirely on-premises. No cloud APIs, no data leaving the network, no ongoing inference costs.
The stack handles natural language queries with real-time web search — including live weather, current news, and anything else that requires up-to-date information beyond the model's training cutoff.
Node 1 — Inference
- Orange Pi 5 Max (RK3588 SoC, 16GB LPDDR5)
- NVMe SSD for stack storage
- Ubuntu (ubuntu-rockchip build), booting from eMMC
Node 2 — Services
- Orange Pi 5 Max (same spec)
- Handles web UI, search, and crawling workloads
- Offloads memory pressure from the inference node
The RK3588's NPU handles model inference natively via rkllama, leaving system RAM available for the supporting stack.
rkllama provides an OpenAI-compatible API endpoint backed by the RK3588 NPU. Models are converted to .rkllm format using the RKLLM toolkit and quantized for the NPU.
Model: Qwen3-4B-Instruct, converted from HuggingFace weights using the RKLLM toolkit on an x86 machine, then deployed to the OP5M.
Real-world inference speed is approximately 20-22 tokens/second. The NPU keeps model weights in its own memory pool, leaving system RAM free for other services.
Key finding: Qwen3-4B-Instruct supports full OpenAI-compatible function/tool calling, verified by direct API test:
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llm",
"messages": [{"role": "user", "content": "What is the weather in Salt Lake City?"}],
"tools": [{"type": "function", "function": {"name": "web_search", ...}}],
"tool_choice": "auto"
}'
# Returns finish_reason: "tool_calls" ✅LiteLLM acts as a model router, providing a unified API endpoint across multiple inference nodes. Configured with least-busy routing and Redis-backed caching.
model_list:
- model_name: home-cluster-llm
litellm_params:
api_base: http://rkllama:8080/v1
custom_llm_provider: openai
model: llm
api_key: dummy-key
router_settings:
routing_strategy: least-busy
redis_url: "redis://redis:6379"LiteLLM earns its place when multiple inference nodes are present. With a single node it's a lightweight pass-through, but the config is ready to uncomment additional nodes as hardware is added.
The critical missing piece in most local LLM setups is the agentic tool loop. LiteLLM and most chat UIs will relay a tool_calls response back to the client, but nothing executes the tool and feeds results back to the model. This custom FastAPI service fills that gap.
The pipeline:
UI → Agent (port 5000) → LiteLLM → rkllama
↓ (tool_calls response)
SearXNG (find URLs)
↓
Crawl4AI (fetch full page content)
↓
rkllama (final answer with real data)
↓
UI (streamed response)
Key implementation details:
- Tools are injected into every request — the model always knows
web_searchis available - Arguments normalisation — rkllama returns tool call arguments as a parsed dict, but the OpenAI spec requires a JSON string. Silent mismatch here caused hard-to-debug failures
- Weather fast-path — wttr.in is queried directly for weather queries, returning structured JSON (temperature, conditions, hourly forecast, thunder/rain percentages) without needing a browser crawl
- Real streaming — the tool loop runs blocking, then the final inference request is streamed token-by-token to the UI
- System prompt injection —
/no_thinkdirective disables Qwen3's chain-of-thought reasoning for tool call decisions, significantly reducing latency. Current date is injected to prevent the model reasoning from its training cutoff
messages = [{
"role": "system",
"content": (
f"/no_think "
f"Today is {datetime.now(timezone.utc).strftime('%A, %B %d, %Y')} (UTC). "
"You are a concise assistant with access to a web_search tool. "
"ALWAYS use web_search for current information: weather, news, prices, sports scores. "
"Never refuse or deflect these requests — search first, then answer. "
"After searching, answer in 2-3 sentences using the results. "
"No bullet points or markdown unless explicitly asked."
)
}] + messagesSelf-hosted privacy metasearch engine. Configured with engines that work reliably over VPN/datacenter IPs:
engines:
- name: bing
active: true # most VPN-tolerant
- name: mojeek
active: true # independent index, no blocks
- name: qwant
active: true # European, VPN friendly
- name: wttr
active: true # dedicated weather engine
- name: wikipedia
active: true
- name: google
active: false # blocks datacenter IPs
- name: startpage
active: false # captchas on VPN IPsJSON format must be explicitly enabled in settings.yml:
search:
formats:
- html
- json # required for API accessFor non-weather queries, SearXNG returns URLs with brief snippets — not enough for the model to answer accurately. Crawl4AI fetches the full page content using a headless Chromium browser, giving the model real data to work from.
The agent crawls the top 2 results per query, trims content to 1000 characters per page (sufficient for most queries, keeps context manageable for a 4B model), and falls back to snippets if crawling fails.
Critical API detail: Crawl4AI requires URLs as a list:
payload = {"urls": [url], "priority": 8} # not "urls": urlOpen WebUI connects to the agent wrapper rather than LiteLLM directly, giving it access to the full tool loop. Configured with native SearXNG integration as a secondary search path.
environment:
- OPENAI_API_BASE_URL=http://agent:5000/v1
- ENABLE_WEB_SEARCH=true
- WEB_SEARCH_ENGINE=searxng
- SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>All services communicate over a shared Docker bridge network (ai-bridge) in addition to the default network. This allows service discovery by container name while keeping the stack isolated.
ai-bridge network:
rkllama, litellm, agent, searxng, crawl4ai, open-webui, redis
Critical gotcha: LiteLLM must be explicitly added to ai-bridge — it defaults to only the default network and cannot reach SearXNG or Crawl4AI by container name without it.
With both OP5Ms running, services split by resource profile:
Node 1 — Inference focus
rkllama ← NPU inference (~5GB RAM for 4B model)
litellm ← lightweight router
agent ← FastAPI tool loop (~40MB)
redis ← cache (~5MB)
Node 2 — Services
open-webui ← web UI (~667MB)
searxng ← search aggregator (~112MB)
crawl4ai ← headless browser (~280MB+)
gluetun ← VPN client (optional)
Node 1's agent points at Node 2 for search and crawling:
- SEARXNG_BASE_URL=http://<node2-ip>:8081
- CRAWL4AI_BASE_URL=http://<node2-ip>:11235With the full stack running on a single OP5M (16GB):
| Metric | Value |
|---|---|
| Baseline inference (10 tokens) | ~2.7s |
| Real tokens/second | ~22 tok/s |
| Weather query end-to-end | ~24s |
| General search query | ~60-90s |
| RAM used (full single-node stack) | ~7.5GB |
Memory management is critical. The OP5M has 16GB but with the full stack running, available RAM drops to ~1GB before swap. Adding 8GB swap on the NVMe is essential:
sudo fallocate -l 8G /mnt/ssd/swapfile
sudo chmod 600 /mnt/ssd/swapfile
sudo mkswap /mnt/ssd/swapfile
sudo swapon /mnt/ssd/swapfile
echo '/mnt/ssd/swapfile none swap sw 0 0' | sudo tee -a /etc/fstabThe two-node split resolves memory pressure more cleanly than swap alone.
Tool calling requires an agent loop. LiteLLM and chat UIs relay tool_calls responses but don't execute them. A middleware layer is required to close the loop — receive the tool call, execute the search, inject results, get the final answer.
Search snippets are not enough. SearXNG returns brief page snippets that give the model just enough information to deflect ("check weather.com") rather than answer. Full page content via Crawl4AI gives the model real data.
Engine selection matters. Google, Brave, and Startpage block or captcha datacenter/VPN IPs. Bing, Mojeek, and Qwant are more reliable. wttr.in handles weather directly with structured JSON.
Arguments normalisation is a silent killer. rkllama returns tool call arguments as a parsed Python dict. The OpenAI spec requires a JSON string. Passing the dict directly causes the second-round request to fail silently — the model never sees the search results.
/no_think is significant. Qwen3's chain-of-thought reasoning adds hundreds of tokens before a tool call decision. Disabling it via system prompt cuts the tool decision round from potentially minutes to seconds.
Memory limits on containers matter — except rkllama. Setting memory limits on Open WebUI, LiteLLM, and other services prevents them from consuming RAM that rkllama needs for inference. Do not cap rkllama itself — it needs headroom for context spikes.
Full compose file covers: rkllama, redis, litellm, agent (custom build), open-webui, gluetun, crawl4ai, searxng, with shared ai-bridge network.
The agent wrapper is a custom FastAPI service built from a local Dockerfile. Source available on request.
- Second OP5M node running Node 2 services or starting the service stack elsewhere.
- Qwen2.5-Coder-1.5B on Node 2 for code-focused queries
- FlareSolverr for Crawl4AI anti-bot bypass