Overview of the Issue
MoveTables create does not validate that the source and target keyspaces differ. SwitchTraffic does (via validateMoveTablesSourceKeyspace), but only after the workflow already exists. As a result, a workflow created with --source-keyspace X --target-keyspace X on the same table:
- succeeds at create time (as long as the target table is empty),
- can never be completed (
switchtraffic rejects it with FAILED_PRECONDITION), and
- when cancelled with
--keep-data=false, issues DROP TABLE against what is physically the source table, deleting source data.
The guard that rejects source == target already exists, but is wired only into the SwitchTraffic path, not into create. Validating it at create would make this whole failure mode unreachable.
First reported by @chrisplim.
Reproduction Steps
- In an unsharded keyspace
ks, start with an empty table t.
- Create a MoveTables workflow with source == target:
vtctldclient MoveTables --workflow w --target-keyspace ks create --source-keyspace ks --tables t
Result: the workflow is created successfully.
- Try to complete it:
vtctldclient MoveTables --workflow w --target-keyspace ks switchtraffic
Result: rpc error: code = FailedPrecondition desc = workflow ks.w is invalid: MoveTables source keyspace matches target keyspace (ks). The workflow can never be switched.
- Cancel it:
vtctldclient MoveTables --workflow w --target-keyspace ks cancel --keep-data=false
Result: cancel issues drop table ks.t. Because source and target are the same keyspace and table, this drops the source table.
Note: if t is non-empty, create instead fails the "all target tables are empty" check, so the trap only springs when the target table is empty.
Root cause (analysis)
- The
source == target guard validateMoveTablesSourceKeyspace (in go/vt/vtctl/workflow/server.go) has a single caller: WorkflowSwitchTraffic.
moveTablesCreate (same file) performs no equivalent check.
- The cancel path is
WorkflowDelete -> dropTargets -> removeTargetTables -> drop table <db>.<table> (in go/vt/vtctl/workflow/traffic_switcher.go). The target DB name is derived per keyspace (TabletDbName, go/vt/topo/topoproto/tablet.go), so when the source and target keyspaces are identical, the dropped table is the source.
Proposed fix
Invoke the existing source == target validation (or an equivalent returning the same FAILED_PRECONDITION) at the start of moveTablesCreate, so the degenerate workflow can never be created. Scope to create only; cancel's drop behavior is correct for normal workflows.
Binary Version
Vitess v22 (release-22.0)
Operating System and Environment details
Logic bug, environment-independent. Reproduced on a standard Vitess cluster with a single unsharded keyspace. Not OS- or architecture-specific.
Log Fragments
# switchtraffic on the same-keyspace workflow:
rpc error: code = FailedPrecondition desc = workflow ks.w is invalid: MoveTables source keyspace matches target keyspace (ks)
# cancel --keep-data=false then drops the source table:
drop table ks.t
Overview of the Issue
MoveTablescreatedoes not validate that the source and target keyspaces differ.SwitchTrafficdoes (viavalidateMoveTablesSourceKeyspace), but only after the workflow already exists. As a result, a workflow created with--source-keyspace X --target-keyspace Xon the same table:switchtrafficrejects it withFAILED_PRECONDITION), and--keep-data=false, issuesDROP TABLEagainst what is physically the source table, deleting source data.The guard that rejects
source == targetalready exists, but is wired only into theSwitchTrafficpath, not intocreate. Validating it atcreatewould make this whole failure mode unreachable.First reported by @chrisplim.
Reproduction Steps
ks, start with an empty tablet.rpc error: code = FailedPrecondition desc = workflow ks.w is invalid: MoveTables source keyspace matches target keyspace (ks). The workflow can never be switched.drop table ks.t. Because source and target are the same keyspace and table, this drops the source table.Note: if
tis non-empty,createinstead fails the "all target tables are empty" check, so the trap only springs when the target table is empty.Root cause (analysis)
source == targetguardvalidateMoveTablesSourceKeyspace(ingo/vt/vtctl/workflow/server.go) has a single caller:WorkflowSwitchTraffic.moveTablesCreate(same file) performs no equivalent check.WorkflowDelete->dropTargets->removeTargetTables->drop table <db>.<table>(ingo/vt/vtctl/workflow/traffic_switcher.go). The target DB name is derived per keyspace (TabletDbName,go/vt/topo/topoproto/tablet.go), so when the source and target keyspaces are identical, the dropped table is the source.Proposed fix
Invoke the existing
source == targetvalidation (or an equivalent returning the sameFAILED_PRECONDITION) at the start ofmoveTablesCreate, so the degenerate workflow can never be created. Scope tocreateonly;cancel's drop behavior is correct for normal workflows.Binary Version
Operating System and Environment details
Log Fragments