Skip to content

feat: add substreams sink postgres and substreams sink clickhouse CLI subcommands#761

Open
maoueh wants to merge 80 commits into
developfrom
agents-split-substreams-sink-commands
Open

feat: add substreams sink postgres and substreams sink clickhouse CLI subcommands#761
maoueh wants to merge 80 commits into
developfrom
agents-split-substreams-sink-commands

Conversation

@maoueh
Copy link
Copy Markdown
Contributor

@maoueh maoueh commented May 1, 2026

Summary

Implements two new CLI subcommands:

  • substreams sink postgres — streams data into a PostgreSQL database
  • substreams sink clickhouse — streams data into a ClickHouse database

Both commands support two modes:

  1. DatabaseChanges mode — uses substreams-sink-database-changes proto messages to upsert/delete rows
  2. Proto-based mode — inserts protobuf messages directly, inferring schema from proto descriptors

Changes

New files

  • cmd/substreams/sink_postgres.go — CLI command for Postgres sink
  • cmd/substreams/sink_clickhouse.go — CLI command for ClickHouse sink
  • sink/sql/ — Vendored source packages copied from streamingfast/substreams-sink-sql with import paths rewritten to github.com/streamingfast/substreams/sink/sql/...
    • sink/sql/bytes/ — bytes encoding helpers
    • sink/sql/db_changes/db/ — database loader/dialect (Postgres + ClickHouse)
    • sink/sql/db_changes/bundler/ — file bundler for batch uploads
    • sink/sql/db_changes/sinker/ — sinker implementation using db-changes proto
    • sink/sql/db_proto/ — sinker implementation using proto messages directly
    • sink/sql/pb/ — protobuf generated code for schema/services
    • sink/sql/shared.go — shared service extraction helper

Modified files

  • go.mod / go.sum — added 9 new direct dependencies:
    • github.com/AfterShip/clickhouse-sql-parser v0.4.9
    • github.com/ClickHouse/ch-go v0.68.0
    • github.com/ClickHouse/clickhouse-go/v2 v2.40.3
    • github.com/drone/envsubst v1.0.3
    • github.com/jackc/pgx/v4 v4.18.1
    • github.com/jmoiron/sqlx v1.4.0
    • github.com/lib/pq v1.10.9
    • github.com/streamingfast/substreams-sink-database-changes v1.3.2-0.20260110015235-04b544bbecb9
    • github.com/wk8/go-ordered-map/v2 v2.1.7

Notes

  • substreams-sink-sql cannot be imported as a direct dependency (circular — it imports substreams), so source packages are vendored under sink/sql/ with import paths rewritten
  • ClickHouse sink sets HandleReorgs: false (ClickHouse doesn't support reorgs via history table)
  • Postgres sink sets HandleReorgs: true
  • Uses github.com/mr-tron/base58 (already a transitive dep) instead of btcutil/base58

To sync locally

git pull origin agents-split-substreams-sink-commands

maoueh and others added 30 commits May 1, 2026 03:06
@maoueh
feat: vendor sink/sql from substreams-sink-sql@c05b15e (batch 1/8)
49265aa 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: vendor sink/sql from substreams-sink-sql@c05b15e (batch 2/8)
b67ef6a 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: vendor sink/sql from substreams-sink-sql@c05b15e (batch 3/8)
092fbe9 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: vendor sink/sql from substreams-sink-sql@c05b15e (batch 4/8)
b5c00e5 
Fix Quoted() function and push remaining db_changes files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add db_proto sinker, sinker_factory, database, postgres, sc…
a786683 
…hema files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add postgres database and dialect files
5c59d4d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add clickhouse decimal and integer utility files
ba43590 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add clickhouse database, dialect, and types files
829411c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add postgres dialect for db_changes
504eb1b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add clickhouse accumulator inserter
927931f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add pb services proto
0138ec5 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
sink/sql: add pb deprecated proto
2054915 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
Add substreams sink postgres and clickhouse CLI commands
3ef3649 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
Add go.mod direct dependencies for sink/sql packages
e77ab8d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
Add vendored schema.pb.go for sink/sql package
671a468 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add sink postgres and sink clickhouse CLI subcommands
4cb1c49 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: replace btcutil/base58 with mr-tron/base58 (already in go.mod)
6f30f5f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: register postgres driver in db package (mirrors clickhouse regis…
7e65311 
…tration in dialect_clickhouse.go)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: resolve CI compilation errors in sink/sql
8848fe9 
- Add FieldInfo and TableInfo functions to db_proto/proto/utils.go
- Create testing.go with TestTx type in db_changes/db/
- Copy database.pb.go locally to avoid external module dependency
- Update sinker.go import to use local pbdatabase package

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
chore: add local copy of database.pb.go from substreams-sink-database…
6e44e91 
…-changes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 1)
fb1f0dd 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 2)
59b1f36 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 3)
543977d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 4)
4f1f0e3 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 5 - db …
e807f92 
…dialects and sinker)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 6 - pos…
c646cc8 
…tgres dialect)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: embed substreams-sink-sql source under sink/sql/ (batch 7 - cli…
a3fbf1e 
…ck_house and postgres db_proto)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: push pb files to correct import path locations
6cef49b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: push deprecated.pb.go to correct import path location
b6b29f7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: push database.pb.go to correct import path location
fb8fbf2 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
maoueh and others added 30 commits May 1, 2026 09:35
@maoueh
feat: add db_changes/bundler/writer/{common,interface,types}.go
1d11e1b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/bundler/{bundler,encoder,stats}.go
a3af805 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/bundler/writer/buffered.go
5cecd92 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/sinker/sinker.go
bb6d829 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/sinker/setup.go
fef1548 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/state/{file,interface}.go
0b10979 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/db/dsn.go
fb1419f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/db/{dialect_postgres,dialect_clickhouse}.go
e53b4f6 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add inlined pbdatabase from substreams-sink-database-changes
7cb11aa 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add deprecated pb from substreams-sink-sql
68b5eb6 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add services pb from substreams-sink-sql
bfa36ab 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add schema pb from substreams-sink-sql
ff91d73 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add pb stub files for sink sql packages
ce36704 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add sink/sql root, bytes, proto and db_proto root files
bf1f8df 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_proto sub-packages (proto utils, stats, sql core)
b173bb0 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_proto/sql/schema and click_house types/decimal files
84993b5 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add click_house integer, database files
a5bcd63 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add click_house dialect and accumulator_inserter files
103a385 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add postgres dialect, database, types, inserter files
42e40fb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/db core files (db, dsn, cursor, dialect, flush, …
f6350e3 
…testtx)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/db remaining files (metrics, operations, types, …
6ce688b 
…ops, user)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/db dialect files (clickhouse and postgres)
89b8ba1 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add db_changes/sinker files (sinker, factory, setup, metrics, s…
f318cdc 
…tats)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: remove duplicate TestTx declaration causing compile errors
fa69897 
Delete testing.go which redeclares TestTx already defined in testtx.go.
This was incorrectly included by a previous agent and is not needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: remove unused services files causing compile errors
babe7a1 
The sink/sql/services/ package references proto types not in stub.
These files were incorrectly added and are not needed by the CLI commands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
fix: remove dbt.go services file (unused, references missing proto ty…
479bc7b 
…pes)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
chore: remove misplaced database.pb.go from wrong directory
ceb5fdd 
This file was incorrectly placed under sink/sql/db_changes/pb/...
The correct location is pb/sf/substreams/sink/database/v1/ which
already has a stub.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
chore: remove unused deprecated.pb.go from sink/sql/v1
91893c8 
This file is not referenced by any code in the repository.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add sink/sql db_proto core files from substreams-sink-sql
8486fe0 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maoueh
feat: add sink/sql db_proto sinker and sinker_factory
85dca3c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@maoueh