Skip to content

Improve large sync resumability#92

Merged
nickvanw merged 7 commits into
mainfrom
improve-large-sync-resumability
Jun 17, 2026
Merged

Improve large sync resumability#92
nickvanw merged 7 commits into
mainfrom
improve-large-sync-resumability

Conversation

@nickvanw

@nickvanw nickvanw commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • checkpoint historical copy progress using LastKnownPk cursors
  • preserve the VGTID position on copy-resume cursors so Vitess can run catchup before copying pk > LastKnownPk
  • switch the connector read and cursor-peek paths from psdbconnect.Sync to direct vtgateservice.Vitess/VStream
  • decode raw VStream FIELD, ROW, VGTID, LASTPK, and COPY_COMPLETED events in the connector, and reject copy cursors that lack LastKnownPk field metadata
  • avoid truncating when state already has historical copy progress
  • fail fast when record/truncate/schema-build sends fail so state cannot advance past data Fivetran did not receive
  • propagate Update stream context through schema/shard/sync work and cancellation-aware timeout backoff

Testing

  • GOCACHE=/tmp/go-build go test ./...
  • GOCACHE=/tmp/go-build go vet ./...
  • DATABASE_* go test -tags=integration -run TestIntegrationInterruptedCopyResume -count=1 -timeout 20m -v ./cmd/internal/server/handlers
  • Earlier on this branch: GOCACHE=/tmp/go-build go test -race ./...
  • Earlier on this branch: DATABASE_* INTEGRATION_STRESS=1 go test -tags=integration -run TestIntegration -count=1 -timeout 20m -v ./cmd/internal/server/handlers

@nickvanw nickvanw merged commit 7d3c656 into main Jun 17, 2026
8 checks passed
@nickvanw nickvanw deleted the improve-large-sync-resumability branch June 17, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants