Skip to content

mujib77/rift

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Rift

CI Version License Go

PostgreSQL CDC pipeline in a single Go binary. No Kafka. No JVM. No complexity.


The problem

Syncing data out of Postgres usually means:

Postgres → Debezium → Kafka → Kafka Connect → Destination

Four systems to run, monitor, and debug. Most small teams don't need that complexity.

Rift

Postgres → Rift → Destination

One binary. One config file.


How it works

Rift connects to Postgres using logical replication and streams every INSERT, UPDATE, DELETE to your destinations in real time.

If a destination goes down, Rift writes events to a local BoltDB disk queue and automatically drains them when the destination recovers.

No Kafka needed for that resilience.


Quickstart

1. Enable logical replication

In postgresql.conf:

wal_level = logical

2. Create rift.yaml

source:
  type: postgres
  url: postgres://user:pass@localhost:5432/mydb?replication=database
  slot: rift_slot
  publication: rift_pub

destinations:
  - name: my-webhook
    type: webhook
    url: https://myapp.com/webhook

  - name: analytics-db
    type: postgres
    url: postgres://user:pass@analytics:5432/analytics

  - name: cache
    type: redis
    url: redis://localhost:6379

queue:
  enabled: true
  path: ./rift-queue
  max_size_mb: 1000

filter:
  script: ./filter.js

3. Run

go run main.go

JS Filtering

Drop events at source before they consume bandwidth:

function filter(event) {
  // only sync enterprise users
  if (event.data.plan !== 'enterprise') return false

  // drop test emails
  if (event.data.email.includes('test@')) return false

  return true
}

Point to your script in rift.yaml:

filter:
  script: ./filter.js

Destinations

Type Status Description
Webhook ✅ v0.1.0 HTTP POST with JSON payload
HTTP ✅ v0.1.0 Generic HTTP endpoint
Postgres ✅ v0.3.0 Real-time DB to DB sync
Redis ✅ v0.4.0 Pub/sub + event list

Disk Queue

When a destination goes offline Rift switches to air-gap mode automatically.

Events write to local BoltDB instead of being dropped. When the destination recovers, Rift drains the queue and resumes. No events lost. No manual intervention.


Event format

{
  "table": "users",
  "operation": "INSERT",
  "data": {
    "id": "1",
    "name": "Mujib",
    "email": "mujib@example.com"
  },
  "lsn": "0/16C752F8",
  "timestamp": "2026-05-20T14:32:00Z"
}

Roadmap

v0.1.0  ✅  WAL reading + webhook destination
v0.2.0  ✅  Embedded BoltDB disk queue
v0.3.0  ✅  Postgres destination
v0.4.0  ✅  Redis destination + JS filtering
v0.5.0  →   DDL schema tracking
v1.0.0  →   Production ready — Debezium alternative

Requirements

  • PostgreSQL 12+ with wal_level = logical
  • Go 1.26+

License

MIT

About

Production-grade PostgreSQL CDC pipeline in a single Go binary - streams every INSERT, UPDATE, DELETE to webhooks with embedded disk queue for Kafka-free resilience

Topics

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors