[persist] Compatibility versioning

[bkirwi]

This PR:

• changes the semantics of the Persist shard's version field... instead of indicating the highest version that has ever touched a bit of state, it indicates the lowest version which is guaranteed to be compatible with the existing state.
• changes the version checks: we now disallow versions from the future, and cap the number of versions from the past we will maintain write compatibility with.
• moves the version field into StateCollections, to make it available to individual Persist commands. This will allow future Persist changes to check the version before making changes, which is essential for flexible backwards compatibility.
• adds a new command to upgrade the compatibility version of the shard. This is not yet invoked everywhere it should be; I intend to add the full set of usages in a followup PR.
• adds metrics that report whether we're interacting with an older version of state or not, which (among other things) will help me track whether we're reliably upgrading shards or not.

Motivation

https://github.com/MaterializeInc/database-issues/issues/9870

[teskje]

This looks good to me, at least as far as I'm grokking the persist details. There is still the piece missing where we call upgrade_version on all existing shards in the environment during leader startup, right?

[def-]

I think this Cargo test failure might be caused by this PR? At least I haven't seen it before: https://buildkite.com/materialize/test/builds/111438:

        FAIL [   4.451s] mz-catalog::open test_persist_open
thread 'test_persist_open' panicked at src/catalog/tests/open.rs:549:9:
assertion `left == right` failed

[bkirwi]

I think this Cargo test failure might be caused by this PR?

Probably related and probably not concerning - looks like a snapshot test that doesn't fully reflect the state of the merged branch - but thanks for linking; will keep an eye on it!

[bkirwi]

This doesn't yet include all the necessary upgrade_version calls, but it seems to contain enough to make CI happy. Hunting down everywhere that we open a shard may take a while, though, so I'd love to get the first batch merged and unblock some other folks.

[teskje]

LGTM! Might want a second review from someone with more persist knowledge.

[DAlperin]

This all looks fine to me. I have one question and I don't think we are using the new apis everywhere we need to but you mentioned thats for a followup PR so I'm fine with that. These are exciting changes!

[DAlperin]

Why can't we pass the applier_version here? That is set from the build version, no? In many cases they will be equal but will diverge during upgrades.

[bkirwi]

This is just a field move - can be seen most clearly in the individual commit.

(And yeah, generally the applier version would match the build version, but not always - eg. if we've just loaded up a state from an old version and haven't modified it yet. I will add fixing this fixme to the list of things I'll try and get in for Thursday...)