feat: add PostgreSQL + pgvector backend for shared multi-agent QMD deployments#342
Open
chrisdietr wants to merge 5 commits intotobi:mainfrom
Open
feat: add PostgreSQL + pgvector backend for shared multi-agent QMD deployments#342chrisdietr wants to merge 5 commits intotobi:mainfrom
chrisdietr wants to merge 5 commits intotobi:mainfrom
Conversation
- src/pg.ts: PostgreSQL adapter (Worker thread + Atomics.wait for sync API) - src/pg-worker.ts: Worker thread for async postgres queries - src/db.ts: Backend selection via QMD_BACKEND env (default 'sqlite') - src/store.ts: Backend-aware queries (vec0→pgvector HNSW, FTS5→tsvector+GIN) - test/store.postgres.test.ts: Integration tests (gated by QMD_ENABLE_POSTGRES_TESTS) - README.md: Backend documentation All 28 existing tests pass. 3 new Postgres integration tests pass. Zero breaking changes to SQLite behavior. Built by gpt-5.3-codex (xhigh reasoning, 154K tokens)
SQLite allows non-aggregated columns in GROUP BY; Postgres does not. Wrap c.doc in MIN() for Postgres path to satisfy ANSI GROUP BY rules.
When QMD_BACKEND=postgres, status now shows: - Backend: PostgreSQL (dbname) instead of Index: /path/to/sqlite - Size from pg_database_size() instead of file stat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds PostgreSQL + pgvector as an opt-in storage backend for QMD.
SQLite remains the default backend and continues to be the right choice for
local, single-user workflows. This change is specifically aimed at shared and
multi-agent deployments, where multiple processes may need concurrent access to
the same index and SQLite becomes a limiting operational choice.
The goal is to extend QMD to that environment without changing the default
experience for existing users.
Motivation
QMD works very well today as a local-first tool. But multi-agent setups place
different demands on the storage layer than single-user local use.
When several agents or processes need to share the same memory/index and perform
concurrent reads and writes, a server-backed database is a better fit than
SQLite. PostgreSQL provides a more robust concurrency model while still letting
QMD preserve the same indexing, search, and retrieval model it already has.
In short:
What changed
Backend selection
QMD_BACKENDto select the storage backendQMD_POSTGRES_URLfor PostgreSQL connection configurationDatabase abstraction
PostgreSQL support
tsvectorCLI / status output
qmd statusso it reports PostgreSQL backend information correctlyinstead of assuming a SQLite file path
Documentation
README.mdTests
that validates batching behavior without depending on machine timing
Compatibility
This change is intended to be backward-compatible.
Existing users
PostgreSQL users
PostgreSQL support is opt-in:
Vector handling
Internal vector handling remains aligned with the existing sqlite-vec path.
Embeddings were already represented at the storage boundary as
Float32Array;the PostgreSQL backend follows the same convention and converts values as needed
for pgvector.
Testing
Default suite:
Result:
The 3 skipped tests are the opt-in PostgreSQL integration tests in
test/store.postgres.test.ts.PostgreSQL integration suite:
QMD_ENABLE_POSTGRES_TESTS=1 bun test --preload ./src/test-preload.ts test/store.postgres.test.tsResult:
Note: the PostgreSQL integration test provisions the test database/schema inside
an existing PostgreSQL server, but does not start PostgreSQL itself. It assumes
a reachable Postgres instance with the
vectorextension available.Notes for reviewers
setup with pgvector available
change; I kept it in this PR because it was necessary to make the branch
reliably green
Tooling note
This PR was prepared with pi using GPT-5.4 xhigh.