fix: exit non-zero on partial migration and print reconciliation summary - #27
Open
FedorenkoKirill wants to merge 1 commit into
Open
Conversation
Before this patch migrate.py exits 0 even when INSERTs into PostgreSQL fail: process_table locally tracks failed_rows but nothing bubbles up, and the top-level always prints "Migration Complete!". This masks partial data loss - e.g. when a batch hits a FK-violation the whole pg transaction gets aborted and even previously-inserted rows in that batch are rolled back at commit, so the source and target row counts end up different with no signal to the user. Collect per-table results from process_table, cross-check against SELECT COUNT(*) in Postgres for each migrated table, and print a Rich summary showing source rows, target rows and failed inserts. If any table has a source/target mismatch or any failed inserts, exit 1 instead of the previous silent success.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
migrate.pyexits 0 even when INSERTs into PostgreSQL fail:process_table()tracks failed rows locally and prints per-row errormessages, but nothing bubbles up to the top level, which always prints
Migration Complete!and returns success.This masks partial data loss. In the worst case a batch hits a
FK-violation on its first INSERT, poisons the whole pg transaction
(
current transaction is aborted, commands ignored until end of transaction block), and the trailingcommit()on that abortedtransaction is effectively a rollback — so even the rows that
inserted successfully earlier in the batch are gone. The tool still
exits 0.
I hit this on a real Open WebUI migration: three tables (
chat_file,knowledge_file,group_member) landed with 0 rows in Postgres whilethe tool reported success. Only per-table row-count reconciliation
against the source SQLite surfaced the loss.
Change
process_tablenow returnsTableMigrationResult(source_rows, failed_inserts)instead ofNone.SELECT COUNT(*)in Postgres and compare against the source SQLitecount. This is authoritative — the in-memory "inserted" counter can
lie when a transaction gets aborted mid-batch (per above).
Table / SQLite rows / PostgreSQL rows / Failed inserts / Status.sys.exit(1)if any table has a row-count mismatch or any failedinserts.
Panel("Migration Complete!")prints only in thefully-clean case.
Before / After
Reproduced against a local
postgres:16container bootstrapped withthe Open WebUI schema, and a minimal SQLite (2 users, 2 chats, 1 file,
3
chat_filerows — one valid, two referencing deletedfileids andwith no FK declared in SQLite so
PRAGMA foreign_key_checkreturnsclean and integrity-check passes).
Before:
SELECT COUNT(*) FROM chat_filein Postgres → 0 (not even thevalid row survived the aborted commit).
After:
Scope
Deliberately kept narrow — one PR, one idea:
IGNORABLE_SQLITE_FOREIGN_KEY_VIOLATIONSor theFK-orphan skip logic from resolve issues/22 #24. Happy to open a follow-up
generalizing those hardcoded pairs (my real migration had
chat_file→fileorphans that the current hardcoded list wouldn'tcatch).
fix in that area.
process_table's exception path: an unexpectedexception still aborts the whole run with a stack trace, same as
today.