persist: lazy schema registration (2nd try) #34022
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Second attempt at getting lazy write schema registration working. This was previously attempted in #33902 but had to be reverted because it caused type confusion during compaction. Details in Slack.
We believe that the confusion came from batches not having schema IDs set, so compaction would fall back the write handle's schema, which in the case of shard finalization would be a dummy schema. #33995 removes that fallback logic. In this PR we add additional logic that populates the part schema IDs in
compare_and_append_batch, so compaction should always be able to find a correct schema.Motivation
Fixes https://github.com/MaterializeInc/database-issues/issues/9874
Part of https://github.com/MaterializeInc/database-issues/issues/9793
Tips for reviewer
Here is a branch where I test this under a configuration known to tickle the bug (shard finalization with a non-trivial schema): #33992
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.