feat(index): add optional Milvus derived index backend

Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>

Author: zc277584121

QUICKSTART.md

@@ -81,8 +81,9 @@ works.
 
 ## 3. Start the server
 
-Check your file descriptor limit — EverOS opens many LanceDB segment
-files under concurrent search + indexing. Platform defaults:
+With the default LanceDB index backend, check your file descriptor limit —
+EverOS opens many LanceDB segment files under concurrent search + indexing.
+Platform defaults:
 **macOS 256** · **Linux 1024** · **Windows 8192**. If yours is below
 4096, raise it before starting:
 
@@ -258,7 +259,8 @@ This is what makes EverOS different — memory persists as plain Markdown:
 ├── ome.toml                            ← strategy config (hot-reloaded)
 ├── .index/                             ← derived indexes (rebuildable from md)
 │   ├── sqlite/system.db
-│   └── lancedb/
+│   ├── lancedb/                        ← default derived index backend
+│   └── milvus/                         ← Milvus Lite DB when configured
 └── .tmp/
 ```
 

README.md

@@ -39,8 +39,10 @@
 EverOS is a Python library and local-first memory runtime for agents and
 makers. It gives one portable memory layer across coding assistants, apps,
 devices, and workflows from day one. It stores conversations, files, and agent
-trajectories as readable Markdown, then syncs local SQLite and LanceDB indexes
-for fast retrieval and self-evolving reuse.
+trajectories as readable Markdown, then syncs local SQLite and a derived
+vector/BM25 index for fast retrieval and self-evolving reuse. LanceDB is the
+default embedded index backend; Milvus can be enabled when you want the same
+rebuildable index to run on Milvus Lite, Milvus server, or Zilliz Cloud.
 
 <table>
 <tr>
@@ -60,7 +62,7 @@ for fast retrieval and self-evolving reuse.
 </tr>
 <tr>
 <td><strong>Local three-part stack</strong></td>
-<td>✅ Markdown + SQLite + LanceDB; no MongoDB, Elasticsearch, or Redis required</td>
+<td>✅ Markdown + SQLite + embedded LanceDB by default; optional Milvus when configured</td>
 <td>❌ Often depends on managed services, vector DBs, graph DBs, or server stacks</td>
 </tr>
 <tr>
@@ -114,6 +116,12 @@ uv pip install everos
 # or: pip install everos
 ```
 
+To use Milvus as the derived index backend:
+
+```bash
+uv pip install "everos[milvus]"
+```
+
 ### 2. Play With The Demo
 
 Run this before configuring API keys or starting the server:
@@ -642,7 +650,7 @@ Explore stored entities and relationships in a graph interface. Frontend demo; b
 ## Documentation
 
 - [docs/everos-demo.md](docs/everos-demo.md) — Demo scope and TUI source layout
-- [docs/how-memory-works.md](docs/how-memory-works.md) — Markdown, SQLite, LanceDB, and recall flow
+- [docs/how-memory-works.md](docs/how-memory-works.md) — Markdown, SQLite, derived index backends, and recall flow
 - [docs/use-cases.md](docs/use-cases.md) — Full use-case gallery and integration examples
 - [docs/engineering.md](docs/engineering.md) — Contributor engineering reference: build, test, CI, conventions
 - [docs/migration-to-1.0.0.md](docs/migration-to-1.0.0.md) — Legacy API migration notes

config.example.toml

@@ -56,3 +56,19 @@ max_concurrent  = 5
 #
 # [lancedb]
 # read_consistency_seconds = 5.0
+
+# ── Optional Milvus derived index ─────────────────────
+# Markdown remains the source of truth. Set the index backend to Milvus
+# when you want the rebuildable vector/BM25 index to live in Milvus.
+# Leave uri empty for Milvus Lite at <root>/.index/milvus/milvus.db.
+#
+# [index]
+# backend = "milvus"
+#
+# [milvus]
+# uri = ""                         # or "http://localhost:19530"
+# token = ""                       # required for Zilliz Cloud/auth-enabled Milvus
+# db_name = ""
+# consistency_level = "Session"
+# dimension = 1024                 # must match your embedding model output
+# collection_prefix = "everos"

docs/api.md

@@ -84,8 +84,9 @@ bare FastAPI `detail`); see [Errors](#errors).
 
 `/add` and `/flush` write the markdown file (the source of truth)
 **synchronously** — when the call returns with `status: "extracted"`,
-the new entry exists on disk. The LanceDB vector / BM25 / scalar index
-is rebuilt by the in-process **cascade coroutine asynchronously**.
+the new entry exists on disk. The configured vector / BM25 / scalar
+index backend is rebuilt by the in-process **cascade coroutine
+asynchronously**.
 
 That means `/search` and `/get` may not see a record immediately after
 the `/flush` that produced it. Typical sync latency is sub-second, but
@@ -362,7 +363,7 @@ A recursive boolean tree of predicates. Used by `/search.filters` and
 `/get.filters`. The Pydantic envelope only checks the recursive
 combinator shape; field-level validity (which scalar fields are
 filterable, which operators apply, value coercion) runs when the
-node is compiled to a LanceDB `where` clause server-side. Compile
+node is compiled to a backend-specific filter clause server-side. Compile
 errors surface as `422` with the offending field / operator in
 `error.message`.
 
@@ -455,12 +456,12 @@ Examples:
 |---|---|
 | `"keyword"` | BM25 only — pure lexical match, no embedding cost |
 | `"vector"` | Dense vector ANN only — semantic recall, no lexical |
-| `"hybrid"` *(default)* | Reciprocal-rank fuse of BM25 + vector + optional scalar filter in a single LanceDB query |
+| `"hybrid"` *(default)* | Reciprocal-rank fuse of BM25 + vector + optional scalar filter against the configured derived index backend |
 | `"agentic"` | Iterative cluster-path retrieval driven by a cross-encoder rerank loop; higher quality at higher latency / cost |
 
 `"hybrid"` is the default because it balances recall and precision
-with one LanceDB roundtrip. `"agentic"` calls the LLM in a loop and
-should be reserved for offline or background workflows.
+without requiring the agentic loop. `"agentic"` calls the LLM in a loop
+and should be reserved for offline or background workflows.
 
 ### GetMemoryType
 
@@ -596,7 +597,7 @@ scope.
   this `(session_id, app_id, project_id)`, or it was already flushed).
 
 `/flush` is synchronous with respect to markdown persistence: by the
-time the response returns, the new entry is on disk. LanceDB index
+time the response returns, the new entry is on disk. Derived index
 sync is still asynchronous — see
 [Eventual consistency](#eventual-consistency).
 

docs/architecture.md

@@ -17,7 +17,7 @@
 │    + reflection + strategies + get + events            │
 ├──────────────────────────────────────────────────────┤
 │  infra/persistence  (Storage adapters; infra/ may host other adapter types)    │
-│    markdown + sqlite + lancedb                        │
+│    markdown + sqlite + derived index                  │
 └──────────────────────────────────────────────────────┘
 
 Cross-cutting (used by all layers, depends on none):
@@ -68,8 +68,8 @@ layers = [
 └────────────────────────────────────────────────────────────────┘
 
    ┌──────────────┐   ┌──────────────┐   ┌─────────────────┐
-   │   Markdown   │   │   SQLite     │   │    LanceDB      │
-   │  (truth)     │   │  (state)     │   │  (index)        │
+   │   Markdown   │   │   SQLite     │   │ Derived index   │
+   │  (truth)     │   │  (state)     │   │ LanceDB/Milvus  │
    ├──────────────┤   ├──────────────┤   ├─────────────────┤
    │ entries +    │   │ change queue │   │ vector ANN      │
    │ frontmatter  │   │ + state/LSN  │   │ BM25 (Tantivy)  │
@@ -78,7 +78,7 @@ layers = [
    └──────────────┘   └──────────────┘   └─────────────────┘
           │                  │                    │
           ▼                  ▼                    ▼
-    memory-root/         .index/sqlite/      .index/lancedb/
+    memory-root/         .index/sqlite/      .index/<backend>/
    (truth source)       (system data)       (rebuildable)
 ```
 
@@ -101,10 +101,13 @@ External message
    │        │
    ▼        ▼
 4a. SQLite   4b. memory.cascade  (async daemon)
-    audit        watches md → diff entries → LanceDB sync
+    audit        watches md → diff entries → index sync
 ```
 
-**Key guarantee**: md write is strongly consistent (fsync). LanceDB is eventually consistent. LanceDB unavailability does not block response — changes buffer in the SQLite `md_change_state` queue, replayed on recovery.
+**Key guarantee**: md write is strongly consistent (fsync). The derived
+index is eventually consistent. Index backend unavailability does not block
+response — changes buffer in the SQLite `md_change_state` queue, replayed on
+recovery.
 
 ## Read path
 
@@ -115,8 +118,8 @@ User query
 1. service.search
    │
    ▼
-2. memory.search (hybrid)     single LanceDB query =
-                                BM25 + vector ANN + scalar filter
+2. memory.search (hybrid)     BM25 + vector ANN + scalar filter
+                              through the configured index backend
    │
    ▼
 3. (optional) read md         original markdown for context
@@ -139,12 +142,13 @@ extract/
 
 ### `memory/cascade/`
 
-Daemon that watches markdown changes and syncs to LanceDB:
+Daemon that watches markdown changes and syncs to the configured derived
+index backend:
 
 - inotify / FSEvents file watcher (cross-platform via `watchdog`)
 - 500ms debounce
 - Entry-level diff (added / changed / removed)
-- LanceDB single-transaction update (text + vector columns atomic)
+- Per-entry index upsert / delete (text + vector columns update together)
 - LSN-based crash recovery via the SQLite `md_change_state` queue
 - Handlers for all eight business kinds: episode, atomic_fact, foresight,
   user_profile, agent_case, agent_skill, knowledge_document, knowledge_topic
@@ -225,7 +229,7 @@ holding **only memory extraction algorithms**:
 everalgo is:
 
 - **Stateless** — pure functions, no class hierarchy
-- **No I/O** — does not touch md files / LanceDB / SQLite
+- **No I/O** — does not touch md files, derived indexes, or SQLite
 - **No prompts inline** — extractors that accept a prompt-override parameter use the project-supplied value; others use their algo-bundled defaults
 
 This boundary lets everalgo be reused across product forms (this open-source build, EverOS Cloud, OpenClaw plugins, etc.).

docs/cascade_runbook.md

@@ -1,9 +1,13 @@
 # Cascade Runbook
 
-The cascade daemon keeps LanceDB in sync with the markdown files under
-the memory root. Service / entry points only ever write markdown; the
-daemon is the **sole** writer of the LanceDB index. This runbook covers
-the recurring operational questions.
+The cascade daemon keeps the configured derived index in sync with the
+markdown files under the memory root. Service / entry points only ever write
+markdown; the daemon is the **sole** writer of the derived index. This runbook
+covers the recurring operational questions.
+
+Sections that mention LanceDB-specific schemas, index cache, file descriptors,
+or `lance error` messages apply to the default LanceDB backend. Milvus uses the
+same cascade queue with backend-specific collection management.
 
 ## What runs where
 
@@ -13,7 +17,7 @@ providers in order:
 1. **Metrics** — Prometheus collector.
 2. **LLM** — LLM client initialisation.
 3. **SQLite** — system DB + schema (`SQLModel.metadata.create_all`).
-4. **LanceDB** — async connection + schema verification + FTS indexes.
+4. **Derived index** — async connection + schema verification + search indexes.
 5. **Cascade** — watcher + scanner + worker, all in-process tasks.
 6. **OME** — offline memory engine.
 
@@ -23,7 +27,7 @@ The cascade subsystem itself is three independent loops:
 |---|---|---|
 | Watcher | `watchdog` filesystem events (sync thread) | `md_change_state.upsert` per registered kind |
 | Scanner | Periodic walk (`scan_interval_seconds`, default 30 s) | Same — catches changes the watcher missed |
-| Worker | `claim_pending_batch` polling (default 1 s when idle) | Handler dispatch → LanceDB upsert / delete |
+| Worker | `claim_pending_batch` polling (default 1 s when idle) | Handler dispatch → index upsert / delete |
 
 Every loop talks to the same `md_change_state` sqlite table. The
 worker's claim mode (`pending → processing → done/failed`) keeps

docs/cli.md

@@ -3,7 +3,7 @@
 The `everos` command-line entry point covers **setup and operations** —
 generate starter config files (`init`), run the HTTP API server (`server
 start`), inspect effective config (`config show`), and operate the
-md → LanceDB index queue (`cascade`). Hot-path
+md → derived index queue (`cascade`). Hot-path
 business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
 CLI.
 
@@ -32,7 +32,7 @@ everos
 │   └── show [--root PATH]          Show effective configuration
 ├── server
 │   └── start [--host] [--port] [--root] [--reload] [--log-level]   Start the HTTP API server (uvicorn)
-└── cascade [--root PATH]           Inspect / operate the md → LanceDB sync queue
+└── cascade [--root PATH]           Inspect / operate the md → derived index sync queue
     ├── status                      Queue / LSN summary
     ├── sync [PATH]                 Drain the queue now (optional PATH force-enqueues)
     └── fix [--apply]               List failed rows / re-enqueue retryable ones
@@ -43,8 +43,9 @@ Each subcommand lives in its own module under
 registered in `cli/main.py`. The CLI is intentionally small — hot-path
 business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
 CLI; the CLI covers setup (`init`), running the server, and index ops
-(`cascade`). There is no `reindex` command — rebuild by deleting
-`<root>/.index/lancedb` and restarting, or run `everos cascade sync`.
+(`cascade`). There is no `reindex` command — rebuild by deleting the
+selected backend's derived index directory (`<root>/.index/lancedb` or
+`<root>/.index/milvus`) and restarting, or run `everos cascade sync`.
 
 ## `everos server start`
 
@@ -68,8 +69,8 @@ everos server start \
 | `--root` | `EVEROS_ROOT` | `~/.everos` |
 | `--reload` | — | off (use in development) |
 
-Lifespan startup wires the storage backends (SQLite engine + LanceDB
-connection) on app boot; see
+Lifespan startup wires the storage backends (SQLite engine + configured
+derived index backend) on app boot; see
 [`entrypoints/api/lifespans/`](../src/everos/entrypoints/api/lifespans/).
 
 ## Configuration via env vars
@@ -81,7 +82,9 @@ Both CLI and HTTP server read configuration from `pydantic-settings`:
 | `EVEROS_ROOT` | memory-root path (default `~/.everos`) |
 | `EVEROS_MEMORY__TIMEZONE` | `Settings.memory.timezone` (e.g. `Asia/Shanghai`) |
 | `EVEROS_SQLITE__BUSY_TIMEOUT_MS` | `Settings.sqlite.busy_timeout_ms` |
+| `EVEROS_INDEX__BACKEND` | `Settings.index.backend` (`lancedb` or `milvus`) |
 | `EVEROS_LANCEDB__READ_CONSISTENCY_SECONDS` | `Settings.lancedb.read_consistency_seconds` |
+| `EVEROS_MILVUS__URI` | `Settings.milvus.uri` |
 
 Pattern: `EVEROS_<SECTION>__<KEY>` (double underscore = nesting). See
 [`config/settings.py`](../src/everos/config/settings.py).

docs/configuration.md

@@ -99,6 +99,26 @@ everos init --root /data/everos
 | `read_consistency_seconds` | float \| null | `null` | Read consistency interval. `null` = no check, `0` = strict, `>0` = eventual. |
 | `index_cache_size_bytes` | int | `16777216` | Upper bound on LanceDB index cache (16 MB default). |
 
+### `[index]`
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `backend` | `"lancedb"` \| `"milvus"` | `"lancedb"` | Rebuildable vector/BM25 index backend used by cascade, search, and get. Markdown remains the source of truth; SQLite remains the system state store. |
+
+### `[milvus]`
+
+Milvus is optional. Install with `everos[milvus]`, then set
+`[index].backend = "milvus"`.
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `uri` | string \| null | `""` | Empty uses Milvus Lite at `<root>/.index/milvus/milvus.db`. Set to a Milvus server or Zilliz Cloud endpoint to use an external service. |
+| `token` | string \| null | `""` | Token for Zilliz Cloud or auth-enabled Milvus. |
+| `db_name` | string \| null | `""` | Optional Milvus database name. |
+| `consistency_level` | `"Strong"` \| `"Session"` \| `"Bounded"` \| `"Eventually"` | `"Session"` | Milvus consistency level for created collections. |
+| `dimension` | int | `1024` | Dense vector dimension. Must match the configured embedding model output. |
+| `collection_prefix` | string | `"everos"` | Prefix for EverOS Milvus collections. |
+
 ### `[llm]`
 
 | Field | Type | Default | Required | Description |
@@ -228,3 +248,5 @@ Examples:
 | `[llm] api_key = "sk-..."` | `EVEROS_LLM__API_KEY=sk-...` |
 | `[sqlite] busy_timeout_ms = 10000` | `EVEROS_SQLITE__BUSY_TIMEOUT_MS=10000` |
 | `[memory] timezone = "Asia/Tokyo"` | `EVEROS_MEMORY__TIMEZONE=Asia/Tokyo` |
+| `[index] backend = "milvus"` | `EVEROS_INDEX__BACKEND=milvus` |
+| `[milvus] uri = "http://localhost:19530"` | `EVEROS_MILVUS__URI=http://localhost:19530` |