fix: verify and upgrade the stack, real Graph API data, generic gateway, Redis checkpointing

.env.example

@@ -1,14 +1,26 @@
 
-# Azure OpenAI via LiteLLM
-AZURE_OPENAI_API_KEY=""
-AZURE_OPENAI_API_BASE=""
-# Model deployments you created in Azure OpenAI
-AZURE_OPENAI_GPT4O_MINI_DEPLOYMENT=azure/sc-rnd-gpt-4o-mini-01
-AZURE_OPENAI_EMBEDDING_DEPLOYMENT=azure/ai-rnd-text-embedding-ada-002
+# LLM access. Generic OpenAI-API-compatible config -- point this at Azure
+# OpenAI, a self-hosted LiteLLM proxy, OpenAI directly, or anything else that
+# speaks the OpenAI chat-completions wire format. Not tied to any specific
+# deployment name or host.
+LLM_BASE_URL=https://your-endpoint.example.com/v1
+LLM_API_KEY=your-api-key-here
+LLM_MODEL_ID=your-chat-model-deployment-name
+LLM_EMBEDDING_MODEL_ID=your-embedding-model-deployment-name
+
+# Microsoft Graph (mail + calendar). client_id has no default -- it must be
+# an app registration you create yourself. See docs/graph_setup.md.
+# tenant=common allows both personal Microsoft accounts and work/school
+# tenant accounts to sign in through the same registration.
+MS_GRAPH_CLIENT_ID=your-app-registration-client-id
+MS_GRAPH_TENANT=common
 
 # Runtime
 ENV=dev
-DATA_DIR=./mock_data_json
+# Opspilot's own generated state (tasks, follow-ups, reports, audit/usage
+# logs, email-processing ledger) -- NOT mail/calendar content, which is live
+# from Microsoft Graph and never persisted here.
+DATA_DIR=./data
 POLICY_FILE=./governance/policies.json
 
 # Governance overrides (optional)

.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    name: Tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r backend/requirements.txt
+          pip install pytest pytest-asyncio respx
+
+      # No live Graph or Redis credentials exist in CI, and none are needed:
+      # Graph-dependent code paths raise a clear GraphAuthError when
+      # unconfigured (tested directly), the Graph HTTP boundary itself is
+      # stubbed with respx, and the one test needing a reachable Redis skips
+      # itself when none is found on the default port.
+      - name: Run tests
+        run: pytest tests/ -q
+
+  lint:
+    name: Syntax check
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Byte-compile all sources
+        run: python -m compileall -q agents orchestration backend governance memory repos config scripts tests

.gitignore

@@ -32,6 +32,17 @@ Thumbs.db
 .env.local
 .env.*.local
 
+# Microsoft Graph token cache -- contains a live refresh token, never commit
+.graph_token_cache.json
+
+# Opspilot's own generated runtime state (tasks, follow-ups, reports, audit/
+# usage logs, email-processing ledger). Per-deployment data, not source --
+# same treatment as chroma_db/ and episodes/ below: directory structure is
+# tracked via .gitkeep, generated content is not.
+data/*.json
+data/*/*.json
+!data/**/.gitkeep
+
 # Caching & Logs
 .llm_cache/
 *.log

CLAUDE.md

@@ -10,21 +10,43 @@ a governance layer, and cost tracking.
 The governance + cost-tracking layer is the differentiator. Most multi-agent
 demos skip exactly that part.
 
+Mail and calendar are **live Microsoft Graph data**, not mock JSON. See
+"Microsoft Graph integration" below before touching `repos/`.
+
 ## Stack
 
-Python · LangGraph · LiteLLM · ChromaDB (vector memory) · FastAPI · Next.js ·
-Streamlit / NiceGUI
+Python · LangGraph 1.x (Redis-backed checkpointing) · Microsoft Graph (mail,
+calendar) · Generic OpenAI-API-compatible LLM gateway · ChromaDB (vector
+memory) · FastAPI · Next.js
+
+`streamlit` and `nicegui` were pinned in requirements.txt as a supposed
+"second UI stack" but were never imported anywhere in the codebase --
+removed. Next.js (`frontend/`, 9 pages) is the only real UI and is required.
 
 ## Layout
 
 - `agents/` — the six specialized agents (11 modules)
-- `orchestration/` — LangGraph workflows and routing (11 modules)
+- `orchestration/` — LangGraph workflows and routing (12 modules, including
+  `checkpointer.py`, the shared Redis checkpointer factory)
 - `backend/` — FastAPI server and routes (21 modules)
-- `frontend/` — Next.js dashboard
+- `frontend/` — Next.js dashboard. **Has its own static mock data**
+  (`frontend/public/data/*.json`) and its own "uses mock data as fallback"
+  behavior, untouched by this pass — see "What's still open" below.
 - `memory/` — vector store + episodic memory
-- `governance/` — policy enforcement, audit logging, cost management
-- `repos/` — data access layer
-- `chroma_db/`, `episodes/` — persisted state, committed to the repo
+- `governance/` — policy enforcement, audit logging, cost management, the LLM
+  gateway
+- `repos/` — `data_repo.py` (mail/calendar via Graph + Opspilot's own
+  generated-state JSON), `graph_auth.py`, `graph_client.py`
+- `data/` — Opspilot's own generated state (tasks, follow-ups, reports,
+  audit/usage logs, email-processing ledger). Directory structure is tracked
+  via `.gitkeep`; content is gitignored, same treatment as `chroma_db/` and
+  `episodes/` below. **This is not mock data** — it's the app's own output,
+  starting empty.
+- `chroma_db/`, `episodes/` — persisted state, `.gitkeep`-tracked
+- `docs/graph_setup.md` — Azure Portal app-registration steps (interactive;
+  only you can do this)
+- `scripts/graph_login.py` — run this once to complete the Graph device-code
+  login
 - `ARCHITECTURE.md`, `CODEBASE_INDEX.md`, `super_graph.md` — good existing docs;
   read `ARCHITECTURE.md` first
 - `print_graph.py` — renders the LangGraph topology
@@ -34,24 +56,169 @@ Streamlit / NiceGUI
 ```bash
 cp .env.example .env
 pip install -r requirements.txt
+
+# One-time: create an Azure AD app registration (docs/graph_setup.md),
+# set MS_GRAPH_CLIENT_ID, then:
+python -m scripts.graph_login
+
+# Needs a reachable Redis (checkpointing) and LLM_BASE_URL/LLM_API_KEY/
+# LLM_MODEL_ID (any OpenAI-API-compatible endpoint) to actually run agents.
 ```
 
-## Known issues
-
-- **Unverified.** 13,747 lines of Python landed in 3 commits on a single day
-  (2026-02-05), with one test file. Nothing here has been shown to run
-  end-to-end. Establishing that is the first task, before any feature work.
-- **Deps are ~2 years stale and this is the blocker:** `langgraph==0.2.45`,
-  `litellm==1.42.6`, `langchain-core==0.3.20`, `streamlit==1.40.1`. LangGraph's
-  API moved substantially after 0.2.x — expect real migration work, not a
-  version bump.
-- No CI, no LICENSE.
-- Two UI stacks present (Next.js frontend *and* Streamlit/NiceGUI deps). Pick one.
-- `chroma_db/` and `episodes/` hold runtime state but only `.gitkeep` is tracked,
-  so the directories exist without the data. That part is set up correctly.
-
-## If you're scaling this
-
-Order: prove it runs → upgrade the LangGraph/LiteLLM stack → per-agent evals →
-drop the second UI → deploy a demo. The architecture is sound; the risk is
-entirely in whether it executes.
+## ✅ Deps upgraded and verified — FIXED
+
+The old CLAUDE.md said "expect real migration work, not a version bump." That
+turned out to be mostly wrong, and it was worth checking rather than assuming:
+
+- **`langgraph` 0.2.45 → 1.x**: the actual API surface this codebase touches
+  (`StateGraph`, `END`, `add_node`/`add_edge`/`set_entry_point`,
+  `compile(checkpointer=...)`, `invoke(state, {"configurable": {"thread_id": ...}})`)
+  is **unchanged** across that jump. Verified by constructing and invoking a
+  graph against the installed 1.x, not by reading a migration guide.
+- **`litellm==1.42.6` was dead weight.** Nothing in this codebase has ever
+  imported the `litellm` package — `governance/litellm_gateway.py` always
+  talked to an OpenAI-API-compatible endpoint directly via the `openai` SDK.
+  Removed rather than upgraded.
+- **`langchain-core` pin removed** — nothing here imports `langchain_core`
+  directly; it arrives transitively via `langgraph`.
+- **`pydantic>=1.10` in `backend/requirements.txt` was a real landmine**: every
+  schema in `backend/models.py` / `agents/schemas.py` is v2-only syntax. Tightened
+  to `>=2.7,<3` in both requirement files.
+
+## ✅ Microsoft Graph integration — mail and calendar are real, not mock
+
+`repos/data_repo.py`'s `inbox()`, `meetings()`, `get_transcript()`, and
+`create_meeting()` are backed by live Microsoft Graph calls
+(`repos/graph_client.py`, auth via `repos/graph_auth.py`), not JSON fixtures.
+**There is no local mail/calendar mock data anywhere in the backend on
+purpose** — see `docs/graph_setup.md` for the one-time app-registration setup
+this requires, and `scripts/graph_login.py` for the login itself.
+
+Design points worth knowing before touching this:
+
+- **`DataRepo` stays the single class every call site already constructs**
+  (`DataRepo()`, 31 call sites). Graph-backing is folded directly into it
+  rather than adding a wrapper class that would need touching every
+  construction site.
+- **Field mapping lives in `_map_message`/`_map_event`** in `data_repo.py`,
+  translating Graph's field names into the shapes `backend/models.py` and the
+  agents already expect. Nothing downstream had to change.
+- **`transcript_file` is reused, not repurposed.** It has always meant "the
+  identifier `get_transcript()` needs." Under Graph that identifier is a Teams
+  join URL, not a filename — the call site
+  (`get_transcript(mtg.get("transcript_file"))`) is unchanged.
+- **Personal Microsoft accounts cannot access meeting transcripts at all** —
+  `OnlineMeetingTranscript.Read.All` isn't grantable to personal accounts under
+  any consent; on a work/school tenant it needs admin consent. `get_transcript`
+  catches the resulting 403/401, logs *why* clearly, and returns `""`.
+  `meeting_agent.py`'s existing fallback (a summary from title + attendees)
+  takes over from there — this needed **zero changes** to `meeting_agent.py`.
+- **`tasks`, `followups`, `eod`/`weekly`, `mom_entries`, `users` stay
+  JSON-backed.** These are Opspilot's own generated output (things the agents
+  produce by processing real mail/calendar), not something Graph provides as
+  input. Don't try to "migrate" these to Graph — there's nothing on the Graph
+  side to migrate them to.
+- **Email processing state (`processed`/`agent_actions`/`agent_category`) is a
+  local ledger keyed by `email_id`** (`data/emails/email_state.json`), merged
+  into `inbox()`'s output. Graph has no concept of "has our agent triaged
+  this," so that state has to live somewhere, and it must never be confused
+  with mail content itself.
+- **`create_meeting()` really schedules a calendar event** (`POST
+  /me/events`) — used by the `schedule_meeting` approved-action path in
+  `governance/approval.py`. It no longer appends to a local fake meetings list.
+- **`add_email_to_inbox`** (a "demo sender portal" feature that injected fake
+  mail into a local JSON store) **was removed outright**, not migrated — it
+  fabricated content, which is exactly what this integration exists to stop
+  doing once the input is a real mailbox.
+
+Sync/async: Graph calls are async (`httpx.AsyncClient`); every existing call
+site (`inbox()`, `meetings()`, etc.) is synchronous. `_run_sync()` bridges
+this — safe from both a plain script and from inside a running event loop
+(FastAPI), at the cost of a thread hop in the nested case. Migrating the 31
+call sites to native async is future work, not done here.
+
+## ✅ LLM gateway — generic, and no longer fabricates responses
+
+`governance/litellm_gateway.py` / `governance/gateway.py` used to have
+**three separate layers of fabricated fallback** — `PolicyGateway`'s own
+simulate, `EnhancedLiteLLMGateway`'s simulate, and `PolicyGateway`
+re-simulating if the gateway raised — all returning literal placeholder
+content (`"decisions": ["Decision 1", "Decision 2"]`) when unconfigured,
+indistinguishable from a real analysis to whatever called it. All three are
+gone.
+
+- Config is now generic: `LLM_BASE_URL` / `LLM_API_KEY` / `LLM_MODEL_ID` /
+  `LLM_EMBEDDING_MODEL_ID`, not tied to any specific proxy host or deployment
+  name (`SETTINGS["llm"]`, was `SETTINGS["models"]` with Azure-specific keys).
+- **Fails at construction, not first call.** Missing config raises
+  `LLMNotConfiguredError` immediately — a misconfigured gateway should fail
+  when it's built, not three steps into a workflow.
+- A real API failure after retries **raises**; it does not return a canned
+  string that reads like an answer.
+
+## ✅ Checkpointing — Redis-backed, not in-process
+
+Every graph that persisted state used `MemorySaver` — in-process only, so a
+restart silently discarded every in-flight workflow. `orchestration/checkpointer.py`
+is now the single shared factory (`get_checkpointer()`), backed by
+`langgraph-checkpoint-redis`. Requires a reachable Redis
+(`docker run -p 6379:6379 redis` for local dev); a missing one is a loud
+`CheckpointerUnavailable`, not a silent fall-through to the exact in-memory
+problem this replaces.
+
+`chat_workflow.py` and `followup_reporting_subgraphs.py` imported
+`MemorySaver` but never actually used it — those were dead imports, now
+removed rather than migrated.
+
+## ✅ The app has been shown to boot — first time, per the old CLAUDE.md
+
+`backend/app.py` failed to import at all: `backend/routes_ai.py` imports
+`from app.smart_chat import SmartChatAgent`, and **that module was never
+committed anywhere in this repository**. One missing class took down the
+entire FastAPI app over three endpoints (`/assistant/start`, `/assistant/chat`,
+the third doesn't touch it) out of several dozen.
+
+The import is now optional; only those three endpoints are affected, and they
+answer **501** naming exactly what's missing. **`SmartChatAgent` itself is
+still not implemented** — there's no spec anywhere to reconstruct it from, and
+building an undefined conversational-agent class wasn't part of this pass.
+`backend/app.py` now imports cleanly and registers 74 routes.
+
+## ✅ Real tests + CI — first time
+
+`tests/comprehensive_quality_tests.py` looked like a test suite but isn't one
+pytest can run: it's a live-server smoke script (`python tests/comprehensive_quality_tests.py`)
+that hits a running backend on `localhost:8002` and uses an LLM as a judge. It
+still needs a running server and a configured LLM to do anything.
+
+New, actually pytest-discoverable: `test_graph_client.py` (HTTP boundary
+stubbed with `respx` — test-boundary mocking, not product mock data),
+`test_data_repo.py`, `test_gateway.py`, `test_checkpointer.py`,
+`test_app_boots.py`. **48 pass, 2 skip** (Redis-required cases, skip
+themselves when no Redis is reachable on 6379). `.github/workflows/ci.yml`
+runs the suite on every push/PR — no live Graph or Redis credentials needed;
+Graph failure paths are tested via stubs, not live calls.
+
+## What's still open
+
+- **`SmartChatAgent` is unimplemented.** Building it is a real feature-design
+  task (no spec exists), not something to reverse-engineer.
+- **The frontend has its own static mock data**
+  (`frontend/public/data/*.json`) and its own "mock data as fallback" comment
+  in `frontend/.env.example` — untouched by this pass. Fixing that is Next.js/
+  TypeScript work, a separate chunk from the backend changes here.
+- **deepagents migration** — discussed, not started. The six hand-rolled
+  ReAct agents (`agents/react_agent.py`) are a strong candidate to replace
+  with deepagents' subagent harness, now that the underlying LangGraph/Redis/
+  Graph-API/LLM-gateway foundation is solid. That's the natural next phase.
+
+Resolved, not open: LICENSE already exists (MIT, root of repo) -- the earlier
+note that it was missing was wrong. "Two UI stacks" resolved to one real
+stack (Next.js) plus two unused pip packages, not a decision to make.
+
+## If you're scaling this further
+
+The foundation (deps, real data, gateway, checkpointing, boot verification,
+tests, CI) is done. Order from here: deepagents migration for the six agents →
+`SmartChatAgent` (if the conversational feature is still wanted) → drop the
+second UI → frontend off its own static mock data → LICENSE.

README.md

@@ -214,13 +214,23 @@ Use `.env.example` as a template for your environment setup.
 Copy `.env.example` to `.env` and configure:
 
 ```bash
-# LLM Configuration
-AZURE_OPENAI_KEY=...
-AZURE_OPENAI_ENDPOINT=...
-AZURE_OPENAI_MODEL=...
-
-# Data Configuration
-DATA_DIR=./data/mock_data_json
+# LLM Configuration -- generic OpenAI-API-compatible; not tied to Azure
+# specifically. See .env.example for the full set.
+LLM_BASE_URL=...
+LLM_API_KEY=...
+LLM_MODEL_ID=...
+
+# Microsoft Graph (mail + calendar are live data, not mock JSON).
+# See docs/graph_setup.md before setting these.
+MS_GRAPH_CLIENT_ID=...
+MS_GRAPH_TENANT=common
+
+# Data Configuration -- Opspilot's own generated state (tasks, follow-ups,
+# reports, audit/usage logs). Mail/calendar are never stored here.
+DATA_DIR=./data
+
+# Checkpointing -- requires a reachable Redis
+REDIS_URL=redis://localhost:6379/0
 
 # Server Configuration
 API_PORT=8002

backend/.env.example

@@ -1,11 +1,16 @@
 # Example .env for backend
 ENV=dev
 BACKEND_API_KEY=dev-unprotected
-AZURE_OPENAI_API_KEY=
-AZURE_OPENAI_API_BASE=
-AZURE_OPENAI_GPT4O_MINI_DEPLOYMENT=
-AZURE_OPENAI_EMBEDDING_DEPLOYMENT=
-DATA_DIR=./data/mock_data_json
+
+# See ../.env.example for the full set of LLM / Graph variables and comments.
+LLM_BASE_URL=
+LLM_API_KEY=
+LLM_MODEL_ID=
+LLM_EMBEDDING_MODEL_ID=
+MS_GRAPH_CLIENT_ID=
+MS_GRAPH_TENANT=common
+
+DATA_DIR=./data
 POLICY_FILE=./governance/policies.json
 DAILY_BUDGET_USD=50
 USE_CHROMADB=0

backend/requirements.txt

@@ -1,6 +1,8 @@
 fastapi>=0.95.0
 uvicorn[standard]>=0.22.0
-pydantic>=1.10
+# >=1.10 allowed a pydantic v1 install, which would break outright: every
+# schema in backend/models.py and agents/schemas.py is v2-only syntax.
+pydantic>=2.7,<3
 anyio
 sse-starlette>=1.0.0
 requests>=2.28.0