[schemas] Connector sync state#362
Open
alanshurafa wants to merge 2 commits into
Open
Conversation
Capture connectors all need the same bookkeeping: a resume cursor, a high-watermark, success/error timestamps, and an error count so a stalled or failing connector can be detected. Writing that table per connector is repetitive and drifts. This adds one generic connector_sync_state table plus begin/success/error RPCs that any connector shares, keyed by (connector, surface, sync_key) — not by any thought, so no thought id appears. Pairs with brain-health-monitoring for alerting on bad connectors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Overlapping connector runs can finish out of order; an older run completing late would overwrite a newer cursor/high_watermark. Add a no-regression guard so completions never move them backward, fix the README cursor-read to fail loud on non-2xx, and add a provenance CTA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
schemas/connector-sync-state/— one generic table that any capture connector can use to remember where it left off and whether its last run failed.Every connector that pulls from an external source on a schedule (email importer, calendar poller, RSS reader, webhook receiver, backfill job) needs the same three things: a resume cursor so the next run doesn't re-read everything, lifecycle timestamps, and a last-error plus error count so a stalled or failing connector can be spotted. Writing that table once per connector gets repetitive and inconsistent. This gives them one shared table and three RPCs that wrap the begin / success / error transitions.
What's in it
public.connector_sync_state— one row per(connector, surface, sync_key). Holdscursor_value,high_watermark, the four lifecycle timestamps,last_error,error_count, and free-formcounters/metadatajsonb.connector_sync_begin(...)— upserts the row and marks the run started.connector_sync_success(...)— advances the cursor, stamps success, resets the error count, merges counters. ANULLcursor leaves the existing one in place.connector_sync_error(...)— stamps the error and increments the count, but leaves the cursor untouched so the next run retries from the last known-good position instead of skipping the failed window.Notes for reviewers
BIGINT id; the canonicalpublic.thoughts.id(a UUID) never appears here, and the schema is standalone — it doesn't touchpublic.thoughtsat all.SECURITY INVOKER, and table/function privileges are granted toservice_roleonly. The table also explicitlyREVOKEs fromPUBLIC, anon, authenticated, so a Supabase project that blanket-grants new tables to its API roles still can't expose this one.CREATE TABLE IF NOT EXISTS,CREATE INDEX IF NOT EXISTS,CREATE OR REPLACE FUNCTION; noDROP/TRUNCATE/unqualifiedDELETE. Safe to re-run.Testing
Applied to a throwaway Postgres 18 and exercised end to end: begin → success advances the cursor and keeps
error_countat 0; begin → error increments the count and preserves the cursor; a later clean run resets the count; aNULL-cursor success preserves the prior cursor; distinct surfaces keep independent rows. Verified the statusCHECK, RLS-on,SECURITY INVOKER, and that grants resolve toservice_roleonly — including under a blanketALTER DEFAULT PRIVILEGES ... GRANT ALL ON TABLES TO anon, authenticated, where the explicit revoke keeps anon/authenticated off the table. Re-applied the migration to confirm idempotency. Markdownlint clean against the repo config.