Skip to content

fix: exit non-zero on partial migration and print reconciliation summary - #27

Open
FedorenkoKirill wants to merge 1 commit into
taylorwilsdon:mainfrom
FedorenkoKirill:fix/migration-summary-exit-code
Open

fix: exit non-zero on partial migration and print reconciliation summary#27
FedorenkoKirill wants to merge 1 commit into
taylorwilsdon:mainfrom
FedorenkoKirill:fix/migration-summary-exit-code

Conversation

@FedorenkoKirill

Copy link
Copy Markdown

Problem

migrate.py exits 0 even when INSERTs into PostgreSQL fail:
process_table() tracks failed rows locally and prints per-row error
messages, but nothing bubbles up to the top level, which always prints
Migration Complete! and returns success.

This masks partial data loss. In the worst case a batch hits a
FK-violation on its first INSERT, poisons the whole pg transaction
(current transaction is aborted, commands ignored until end of transaction block), and the trailing commit() on that aborted
transaction is effectively a rollback — so even the rows that
inserted successfully earlier in the batch are gone. The tool still
exits 0.

I hit this on a real Open WebUI migration: three tables (chat_file,
knowledge_file, group_member) landed with 0 rows in Postgres while
the tool reported success. Only per-table row-count reconciliation
against the source SQLite surfaced the loss.

Change

  • process_table now returns TableMigrationResult(source_rows, failed_inserts) instead of None.
  • After the migration loop, for each migrated table run
    SELECT COUNT(*) in Postgres and compare against the source SQLite
    count. This is authoritative — the in-memory "inserted" counter can
    lie when a transaction gets aborted mid-batch (per above).
  • Print a Rich summary table with columns Table / SQLite rows / PostgreSQL rows / Failed inserts / Status.
  • sys.exit(1) if any table has a row-count mismatch or any failed
    inserts. Panel("Migration Complete!") prints only in the
    fully-clean case.

Before / After

Reproduced against a local postgres:16 container bootstrapped with
the Open WebUI schema, and a minimal SQLite (2 users, 2 chats, 1 file,
3 chat_file rows — one valid, two referencing deleted file ids and
with no FK declared in SQLite so PRAGMA foreign_key_check returns
clean and integrity-check passes).

Before:

Failed to migrate 2 rows from chat_file
╭─ Migration Complete! ─╮
EXIT CODE: 0

SELECT COUNT(*) FROM chat_file in Postgres → 0 (not even the
valid row survived the aborted commit).

After:

                        Migration Summary
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Table     ┃ SQLite rows ┃ PostgreSQL rows ┃ Failed inserts ┃ Status   ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ user      │           2 │               2 │              0 │ OK       │
│ chat      │           2 │               2 │              0 │ OK       │
│ file      │           1 │               1 │              0 │ OK       │
│ chat_file │           3 │               0 │              2 │ MISMATCH │
└───────────┴─────────────┴─────────────────┴────────────────┴──────────┘
Migration incomplete: source and target row counts differ for 1 table(s): chat_file
EXIT CODE: 1

Scope

Deliberately kept narrow — one PR, one idea:

  • Not touching IGNORABLE_SQLITE_FOREIGN_KEY_VIOLATIONS or the
    FK-orphan skip logic from resolve issues/22 #24. Happy to open a follow-up
    generalizing those hardcoded pairs (my real migration had
    chat_file→file orphans that the current hardcoded list wouldn't
    catch).
  • Not touching table processing order — added migration order to respect FK dependencies #26 already proposes a
    fix in that area.
  • Not changing process_table's exception path: an unexpected
    exception still aborts the whole run with a stack trace, same as
    today.

Before this patch migrate.py exits 0 even when INSERTs into PostgreSQL
fail: process_table locally tracks failed_rows but nothing bubbles up,
and the top-level always prints "Migration Complete!". This masks
partial data loss - e.g. when a batch hits a FK-violation the whole
pg transaction gets aborted and even previously-inserted rows in that
batch are rolled back at commit, so the source and target row counts
end up different with no signal to the user.

Collect per-table results from process_table, cross-check against
SELECT COUNT(*) in Postgres for each migrated table, and print a Rich
summary showing source rows, target rows and failed inserts. If any
table has a source/target mismatch or any failed inserts, exit 1
instead of the previous silent success.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant