Skip to content

AGENTS.md

Latest commit

 

History

History
356 lines (256 loc) · 18.8 KB

AGENTS.md

File metadata and controls

356 lines (256 loc) · 18.8 KB
Raw

AGENTS.md

This file provides guidance to coding agents working with code in this repository. Claude Code compatibility is kept through CLAUDE.md, which imports this file.

Build and Test Commands

This project uses uv for Python project management.

Building

uv run maturin develop   # Build Rust code and install Python package (required after Rust changes)

Testing

cargo test               # Run Rust tests
uv run mypy              # Type check Python code
uv run pytest python/    # Run Python tests (use after both Rust and Python changes)

Code Formatting and Linting

uv run ruff format .           # Format Python code
uv run ruff format --check .   # Check formatting without making changes (same as CI)
uv run ruff check .            # Lint Python code
cargo fmt                      # Format Rust code

Pre-submission Checks

CI enforces the full pre-commit hook suite from .pre-commit-config.yaml, which covers more than the commands above — notably regenerating the CLI docs (docs/src/content/docs/cli.mdx) whenever cli.py changes. Run it locally before pushing:

uv tool install prek     # one-time; some hooks also need `protoc` (brew install protobuf)
prek run --all-files

Workflow Summary

Change Type Commands to Run
Rust code only uv run maturin develop && cargo test
Python code only uv run mypy && uv run pytest python/
Both Rust and Python Run all commands from both categories above
Python formatting uv run ruff format .
Before pushing prek run --all-files

Code Structure

cocoindex/
├── rust/                       # Rust crates (workspace)
│   ├── core/                   # Core engine crate
│   │   └── src/
│   │       ├── engine/         # Core engine
│   │       ├── state/          # States of the core engine
│   │       └── inspect/        # Database inspection utilities
│   ├── py/                     # Python bindings (PyO3)
│   ├── py_utils/               # Python-Rust utility helpers (error, convert, future)
│   ├── utils/                  # General utilities: error, batching, fingerprint, etc.
│   └── ops_text/               # Text processing operations (splitter, language detection)
│
├── python/
│   ├── cocoindex/              # Python package
│   │   ├── __init__.py         # Package entry point
│   │   ├── cli.py              # CLI commands
│   │   ├── _internal/          # Internal implementation for the core engine
│   │   │   ├── api.py          # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop
│   │   │   ├── app.py          # App base implementation
│   │   │   ├── context_keys.py # ContextKey and ContextProvider
│   │   │   ├── environment.py  # Environment and lifespan handling
│   │   │   ├── function.py     # @coco.fn decorator implementation
│   │   │   ├── component_ctx.py # ComponentContext and component_subpath
│   │   │   ├── target_state.py # Target state implementation
│   │   │   └── core.pyi        # Type stubs for the Rust extension module (update when PyO3 APIs change)
│   │   ├── connectors/         # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)
│   │   ├── connectorkits/      # Connector building utilities
│   │   ├── resources/          # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py
│   │   └── ops/                # Operations: text.py (RecursiveSplitter), sentence_transformers.py
│   └── tests/                  # Python tests
│
├── examples/                   # Example applications
├── docs/                       # Documentation
└── dev/                        # Development utilities

Key Concepts

Mental model: declarative data pipelines

CocoIndex uses a declarative programming model — you specify what your output should look like (target states), not how to incrementally update it. The engine handles change detection and applies minimal updates automatically.

Think of it like:

  • React: declare UI as function of state → React re-renders what changed
  • Spreadsheets: declare formulas → cells recompute when inputs change
  • CocoIndex: declare target states as function of source → engine syncs what changed

Core concepts

App — The top-level runnable unit. Bundles a main function with its arguments. When you call app.update(), the main function runs as the root processing component.

Processing Component — The unit of execution that owns a set of target states. Created by mount() or use_mount() at a specific component path. When a component finishes, its target states sync atomically to external systems.

Component Path — Stable identifier for a processing component across runs. Created via coco.component_subpath("process", filename). CocoIndex uses component paths to:

  • Match components to their previous runs for change detection
  • Determine ownership of target states (if a path disappears, its target states are cleaned up)

Target State — What you want to exist in an external system (a file, a database row, a table). You declare target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.

Target — The API object used to declare target states (e.g., DirTarget, TableTarget). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).

Function — A Python function decorated with @coco.fn. Use memo=True to enable memoization (skip execution when inputs and code are unchanged).

Context — React-style provider mechanism for sharing resources. Define keys with ContextKey[T], provide values in lifespan via builder.provide(), use in functions via coco.use_context(key).

Key APIs

# Mounting processing components (subpath auto-derived from fn.__name__)
await coco.mount(fn, *args, **kw)                                       # child runs independently
result = await coco.use_mount(fn, *args, **kw)                          # returns value directly

# Explicit subpath (for multi-part paths or multiple mounts of same function)
await coco.mount(coco.component_subpath("process", filename), fn, *args, **kw)
result = await coco.use_mount(coco.component_subpath("name"), fn, *args, **kw)

# Component subpath composition
subpath = coco.component_subpath("process", filename)  # multiple parts
subpath = coco.component_subpath("a") / "b" / "c"      # chaining with /

# Using component_subpath as context manager (applies to all nested mount calls)
with coco.component_subpath("process"):
    for f in files:
        await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)

# Declaring target states (typically via Target methods)
dir_target.declare_file(filename=name, content=data)
table_target.declare_row(row=MyRow(...))

# Using context values
db = coco.use_context(PG_DB)  # retrieve value provided in lifespan

# Explicit context management (for ThreadPoolExecutor)
ctx = coco.get_component_context()
with ctx.attach():
    # coco APIs work correctly in this thread
    coco.mount(...)

Mount handles:

  • mount() → ComponentMountHandle: call await handle.ready() to wait until target states are synced
  • use_mount() → returns the result value directly (awaitable)

How syncing works

When a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:

  • New target states → create (insert row, create file)
  • Changed target states → update
  • Missing target states → delete

Changes are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.

Example

import pathlib

import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

@coco.fn(memo=True)
async def process_file(file: FileLike, target: localfs.DirTarget) -> None:
    html = _markdown_it.render(await file.read_text())
    outname = "__".join(file.file_path.path.parts) + ".html"
    target.declare_file(filename=outname, content=html)

@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
    target = await coco.use_mount(localfs.declare_dir_target, outdir)

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
    )
    await coco.mount_each(process_file, files.items(), target)


app = coco.App(
    coco.AppConfig(name="FilesTransform"),
    app_main,
    sourcedir=pathlib.Path("./docs"),
    outdir=pathlib.Path("./out"),
)
app.update_blocking(report_to_stdout=True)

Code Conventions

Internal vs External Modules

We distinguish between internal modules (under packages with _ prefix, e.g. _internal.* or connectors.*._source) and external modules (which users can directly import).

External modules (user-facing, e.g. cocoindex/ops/sentence_transformers.py):

  • Be strict about not leaking implementation details
  • Use __all__ to explicitly list public exports
  • Prefix ALL non-public symbols with _, including:
    • Standard library imports: import threading as _threading, import typing as _typing
    • Third-party imports: import numpy as _np, from numpy.typing import NDArray as _NDArray
    • Internal package imports: from cocoindex.resources import schema as _schema
  • Exception: TYPE_CHECKING imports for type hints don't need prefixing

Internal modules (e.g. cocoindex/_internal/component_ctx.py, **/_target.py):

  • Less strict since users shouldn't import these directly
  • Standard library and internal imports don't need underscore prefix
  • Only prefix symbols that are truly private to the module itself (e.g. _context_var for a module-private ContextVar)

Minimize API surface until deemed necessary

When adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives "in case someone needs them." Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.

General principles (also covered by /review-changes)

  • Top-level imports. Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.
  • Specific types over Any. Use concrete types — including concrete types from third-party libraries. Any only when the type is truly generic and no downstream code needs to downcast it.
  • Validate and exchange early. When a value enters as a weaker form (str, Any, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a ContextKey string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores _pool: asyncpg.Pool, not _db_key: str.
  • NamedTuple/small dataclass for multi-value returns. Access fields by name (result.can_reuse) at call sites.
  • Exceptions for exceptional situations only. Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, "not found", or other expected state transitions. Use explicit return values (sentinel, enum variant, None) for those.
  • Single source of truth, delete dead code, honest names. When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.

Testing Guidelines

We prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.

Test Environment Setup

Use common.create_test_env(__file__) to create a CocoIndex Environment for tests. It derives a unique db_path from the test file path and picks up the current event loop automatically.

  • Sync tests (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.

    coco_env = common.create_test_env(__file__)

  • Async tests with async resources (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the suffix parameter when each test needs its own Environment:

    @pytest_asyncio.fixture
async def pg_env(request: pytest.FixtureRequest) -> Any:
    pool = await asyncpg.create_pool(dsn)
    coco_env = common.create_test_env(__file__, suffix=request.node.name)
    coco_env.context_provider.provide(DB_KEY, pool)
    yield pool, coco_env
    await pool.close()

    The suffix ensures each test gets a unique db_path, avoiding "environment already open" errors.

Testcontainers for Database Tests

Use testcontainers[postgres] (in the build-test dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:

@pytest.fixture(scope="module")
def pg_dsn() -> Any:
    with PostgresContainer("postgres:16-alpine") as pg:
        dsn = pg.get_connection_url().replace("postgresql+psycopg2://", "postgresql://")
        yield dsn

@pytest_asyncio.fixture
async def pool(pg_dsn: str) -> Any:
    p = await asyncpg.create_pool(pg_dsn)
    yield p
    await p.close()

LMDB write paths

  • All LMDB writes must go through Storage::run_txn (uses the single-writer batcher).
  • Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.
  • LMDB has no savepoints. If a sub-operation needs to "abort," handle it at the body level (e.g. return a sentinel result without writing); never attempt per-body rollback inside the batcher.

Sync vs Async

The Rust core (rust/core, rust/utils) uses async-first design with Tokio. The rust/py crate bridges Rust async to Python, offering both sync and async APIs:

  • Rust core exposes async functions
  • rust/py provides sync wrappers that use block_on() to call async Rust from sync Python
  • Python's cocoindex API is async-first: mount APIs (mount, use_mount, mount_each, map) are all async def; App.update()/App.drop() are async; sync entry points (App.update_blocking(), App.drop_blocking(), start_blocking(), stop_blocking()) are available for CLI and blocking contexts

When adding new functionality that involves I/O or concurrency:

  • Implement async in Rust
  • Bridge to Python via rust/py, providing both sync and async variants if needed

Versioning

The current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the main branch; the legacy v0 code is preserved on the v0 branch.

Docs diagrams

All diagrams embedded in docs pages are built as inline Astro components under docs/src/components/diagrams/, not exported SVG files. See docs/src/components/diagrams/README.md for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.

Agent Tooling

Semantic code search over this repo (cocoindex-code)

To find where a concept lives, prefer semantic search over blind grep. This repo keeps a local cocoindex-code index; query it from any agent's shell with the ccc CLI:

ccc search "where is X handled"      # semantic search; add --refresh to reindex first

If ccc search reports no index, set it up once:

command -v ccc >/dev/null || uv tool install 'cocoindex-code[full]'   # per machine; local embeddings, no API key
ccc init -f && ccc index                                             # repo-scoped index (first build ~1-2 min)

Everything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it fresh. On a brand-new machine ccc init asks once for an embedding provider; choose the local default. The -f matters: if a parent directory is also a ccc project, plain ccc init refuses and search silently uses the parent's index, so run ccc status to confirm the root is this repo. We register no MCP server on purpose — an ccc mcp mode exists, but the CLI costs no context until invoked and behaves the same across agents. Source: github.com/cocoindex-io/cocoindex-code.

Agent Playbooks

Reusable task-specific agent guidance lives under dev/agent-skills/. These playbooks use a SKILL.md format so Claude Code can load them directly from .claude/skills/, but the content is intended to be readable by any coding agent.

  • For docs diagrams, read dev/agent-skills/cocoindex-diagrams/SKILL.md.
  • For new target connectors, read dev/agent-skills/target-connector/SKILL.md.
  • For upgrading example package versions, read dev/agent-skills/upgrade-examples/SKILL.md.

Agent Skills-compatible clients, including Codex, discover the checked-in symlinks under .agents/skills/. The public CocoIndex skill is exposed there as cocoindex, and contributor playbooks are exposed there from dev/agent-skills/.

Claude Code compatibility remains under .claude/: hooks/settings live there, and .claude/skills/ mirrors the contributor playbooks with symlinks. Do not duplicate skill content under .agents/skills/ or .claude/skills/; edit the source directories (skills/cocoindex/ for the public skill, dev/agent-skills/ for contributor playbooks) and keep dot-directory entries as symlinks.

Portable agent checks live under dev/agent-checks/. Claude Code hooks in .claude/hooks/ are thin adapters that call these scripts when relevant files changed.