🌌 LLMgine

LLMgine is a pattern-driven framework for building production-grade, tool-augmented LLM applications in Python.
It offers a clean separation between engines (conversation logic), models/providers (LLM back-ends), tools (function calling with MCP support), a streaming message-bus for commands & events, and opt-in observability.
Think FastAPI for web servers or Celery for tasks—LLMgine plays the same role for complex, chat-oriented AI.

---

✨ Feature Highlights

Area	What you get	Key files
Engines	Plug-n-play Engine subclasses (SinglePassEngine, ToolChatEngine, …) with session isolation, tool-loop orchestration, and CLI front-ends	engines/*.py, src/llmgine/llm/engine/
Message Bus	Async command bus (1 handler) + event bus (N listeners) + sessions for scoped handlers	src/llmgine/bus/
Tooling	Declarative function-to-tool registration, multi-provider JSON-schema parsing (OpenAI, Claude, DeepSeek), async execution pipeline, MCP integration for external tool servers	src/llmgine/llm/tools/
Providers / Models	Wrapper classes for OpenAI, OpenRouter, Gemini 2.5 Flash etc. without locking you in	src/llmgine/llm/providers/, src/llmgine/llm/models/
Unified Interface	Single API for OpenAI, Anthropic, and Gemini - switch providers by changing model name	src/llmgine/unified/
Context Management	Simple and in-memory chat history managers, event-emitting for retrieval/update	src/llmgine/llm/context/
UI	Rich-powered interactive CLI (EngineCLI) with live spinners, confirmation prompts, tool result panes	src/llmgine/ui/cli/
Observability	Console + JSONL file handlers, per-event metadata, easy custom sinks	src/llmgine/observability/
Bootstrap	One-liner ApplicationBootstrap that wires logging, bus startup, and observability	src/llmgine/bootstrap.py

---

🏗️ High-Level Architecture

flowchart TD
    %% Nodes
    AppBootstrap["ApplicationBootstrap"]
    Bus["MessageBus<br/>(async loop)"]
    Obs["Observability<br/>Handlers"]
    Eng["Engine(s)"]
    TM["ToolManager"]
    Tools["Local Tools"]
    MCP["MCP Servers"]
    Session["BusSession"]
    CLI["CLI / UI"]

    %% Edges
    AppBootstrap -->|starts| Bus

    Bus -->|events| Obs
    Bus -->|commands| Eng
    Bus -->|events| Session

    Eng -- status --> Bus
    Eng -- tool_calls --> TM

    TM -- executes --> Tools
    TM -. "MCP Protocol" .-> MCP
    Tools -- ToolResult --> CLI
    MCP -. "External Tools" .-> CLI

    Session --> CLI

Loading

Every component communicates only through the bus, so engines, tools, and UIs remain fully decoupled.

---

🚀 Quick Start

1. Install

git clone https://github.com/your-org/llmgine.git
cd llmgine
python -m venv .venv && source .venv/bin/activate
pip install -e ".[openai]"   # extras: openai, openrouter, mcp, dev, …
export OPENAI_API_KEY="sk-…" # or OPENROUTER_API_KEY / GEMINI_API_KEY

2. Run the demo CLI

python -m llmgine.engines.single_pass_engine  # pirate translator
# or
python -m llmgine.engines.tool_chat_engine    # automatic tool loop

You’ll get an interactive prompt with live status updates and tool execution logs.

---

🧑‍💻 Building Your Own Engine

from llmgine.llm.engine.engine import Engine
from llmgine.messages.commands import Command, CommandResult
from llmgine.bus.bus import MessageBus

class MyCommand(Command):
    prompt: str = ""

class MyEngine(Engine):
    def __init__(self, session_id: str):
        self.session_id = session_id
        self.bus = MessageBus()

    async def handle_command(self, cmd: MyCommand) -> CommandResult:
        await self.bus.publish(Status("thinking", session_id=self.session_id))
        # call LLM or custom logic here …
        answer = f"Echo: {cmd.prompt}"
        await self.bus.publish(Status("finished", session_id=self.session_id))
        return CommandResult(success=True, result=answer)

# Wire into CLI
from llmgine.ui.cli.cli import EngineCLI
chat = EngineCLI(session_id="demo")
chat.register_engine(MyEngine("demo"))
chat.register_engine_command(MyCommand, MyEngine("demo").handle_command)
await chat.main()

---

🔧 Tool Integration

Local Tools in 3 Lines

from llmgine.llm.tools.tool import Parameter
from llmgine.engines.tool_chat_engine import ToolChatEngine

def get_weather(city: str):
    """Return current temperature for a city.
    Args:
        city: Name of the city
    """
    return f"{city}: 17 °C"

engine = ToolChatEngine(session_id="demo")
await engine.register_tool(get_weather)               # ← introspection magic ✨

The engine now follows the OpenAI function-calling loop:

User → Engine → LLM (asks to call get_weather) → ToolManager → get_weather()
          ↑                                        ↓
          └───────────    context update   ────────┘ (loops until no tool calls)

MCP (Model Context Protocol) Integration

Connect to external tool servers using the MCP protocol:

from llmgine.llm.tools.tool_manager import ToolManager

# Initialize tool manager with MCP support
tool_manager = ToolManager()

# Register an MCP server (e.g., Notion integration)
await tool_manager.register_mcp_server(
    server_name="notion",
    command="python",
    args=["/path/to/notion_mcp_server.py"]
)

# MCP tools are now available alongside local tools

MCP Features:

• 🔌 External Tool Servers: Connect to any MCP-compatible tool server
• 🔄 Dynamic Loading: Tools are discovered and loaded at runtime
• 🎯 Provider Agnostic: Works with OpenAI, Anthropic, and Gemini tool formats
• ⚡ Graceful Degradation: Falls back silently if MCP dependencies unavailable

Prerequisites:

pip install mcp  # Optional: for MCP integration

---

📰 Message Bus in Depth

from llmgine.bus.bus import MessageBus
from llmgine.bus.session import BusSession

bus = MessageBus()
await bus.start()

class Ping(Command): pass
class Pong(Event): msg: str = "pong!"

async def ping_handler(cmd: Ping):
    await bus.publish(Pong(session_id=cmd.session_id))
    return CommandResult(success=True)

with bus.create_session() as sess:
    sess.register_command_handler(Ping, ping_handler)
    sess.register_event_handler(Pong, lambda e: print(e.msg))
    await sess.execute_with_session(Ping())      # prints “pong!”

Handlers are auto-unregistered when the BusSession exits—no leaks.

---

📊 Observability

Add structured logs with zero boilerplate:

from llmgine.bootstrap import ApplicationBootstrap, ApplicationConfig
config = ApplicationConfig(enable_console_handler=True,
                           enable_file_handler=True,
                           log_level="debug")
await ApplicationBootstrap(config).bootstrap()

The observability system uses a standalone ObservabilityManager that:

• Operates independently of the message bus (no circular dependencies)
• Provides synchronous handlers (see note below)
• Supports console, file, and OpenTelemetry handlers
• Can be extended with custom handlers

All events flow through the configured handlers to console and/or timestamped logs/events_*.jsonl files.

⚠️ Performance Note: The current implementation uses synchronous handlers which can block the event loop in high-throughput scenarios. This is suitable for development and low-volume production use. See the Observability System documentation for details and planned improvements.

For OpenTelemetry support:

pip install llmgine[opentelemetry]

---

📚 Documentation

Comprehensive documentation is available in the docs/ directory:

• Documentation Index - Start here for navigation
• Product Requirements Document - Original project requirements
• Brownfield Enhancement PRD - Production-ready enhancements
• User Stories - Detailed implementation stories
• Engine Development Guide - How to build custom engines
• CLAUDE.md - AI assistant instructions for contributors

Component Documentation

• Message Bus - Event and command bus architecture
• Tools System - Function calling, tool registration, and MCP integration
• Observability System - Standalone observability architecture
• Observability CLI - CLI monitoring tool
• Observability GUI - React-based monitoring interface

---

📁 Repository Layout (abridged)

llmgine/
│
├─ engines/            # Turn-key example engines (single-pass, tool chat, …)
└─ src/llmgine/
   ├─ bus/             # Message bus core + sessions
   ├─ llm/
   │   ├─ context/     # Chat history & context events
   │   ├─ engine/      # Engine base + dummy
   │   ├─ models/      # Provider-agnostic model wrappers
   │   ├─ providers/   # OpenAI, OpenRouter, Gemini, Dummy, …
   │   └─ tools/       # ToolManager, parser, register, types, MCP integration
   ├─ observability/   # Console & file handlers, log events
   └─ ui/cli/          # Rich-based CLI components

---

🏁 Roadmap

• Streaming responses with incremental event dispatch
• WebSocket / FastAPI front-end (drop-in replacement for CLI)
• Persistent vector memory layer behind ContextManager
• Plugin system for third-party Observability handlers
• More providers: Anthropic, Vertex AI, etc.

---

🤝 Contributing

1. Fork & create a feature branch
2. Ensure pre-commit passes (ruff, black, isort, pytest)
3. Open a PR with context + screenshots/GIFs if UI-related

---

📄 License

LLMgine is distributed under the MIT License—see LICENSE for details.

---

“Build architecturally sound LLM apps, not spaghetti code.
Welcome to the engine room.”