feat: crdt for group membership and metadata - #894
Open
dastansam wants to merge 8 commits into
Open
Conversation
dastansam
marked this pull request as ready for review
June 22, 2026 08:54
dastansam
force-pushed
the
feat/group-membership-crdt
branch
from
July 5, 2026 12:25
552041f to
8f62a67
Compare
First slice of the signed-op CRDT that replaces the rev-counter last-writer-wins merge (docs/proposals/Group-State-CRDT.md): a pure, storage/proto-free core so the merge + authorization logic is unit testable in isolation. Design: state is a grow-only set of signed ops (GroupOp), deduped by op_id; merge is set union (commutative/associative/idempotent). The membership/metadata view is DERIVED by a deterministic fold over the ops in total (lamport, actor_id, op_id) order: - Membership = OR-Set keyed by member_id. A Remove tombstones the add op_ids it observed, so a concurrent fresh re-add survives → ADD-WINS on concurrent add-vs-remove (the mesh-invite default; flagged for product review). - Metadata (name, avatar) = per-field LWW register keyed by (lamport, actor_id). Authorization is checked at APPLY time, not merge time — unauthorized ops merge but are skipped by the fold, so a later admin promotion with a lower lamport retroactively legitimises a dependent op with no rebroadcast. v1 policy: any member may Add (open invite; invite_only flips to admins-only); only admins may Remove / UpdateMetadata; the founder is the bootstrap admin, is un-removable, and is the ONLY actor who may remove an admin (per Dastan). Compaction: an admin Compact op raises a lamport floor; ops below it are rejected on merge (tombstone GC within the DTN horizon). 17 unit tests: add/remove, re-add, concurrent add-wins, merge-order independence (convergence), retroactive legitimization, founder-only- removes-admin, admin-removes-member, un-removable founder, invite-only, LWW metadata + admin gate, idempotent merge, compaction floor + admin gate. Builds clean. Follow-ups (next slices): GroupOp protobuf + signing/verify; wire it into Group via Capabilities::GROUP_CRDT with translation to today's invite/reply path for mixed groups; persist the op set; epoch/Compact RPC surface.
Second slice of the group CRDT (docs/proposals/Group-State-CRDT.md):
the signed-op wire format and the codec bridging it to the proto-free
core.
- group_net.proto: GroupOp { group_id, op_id, actor_id, lamport,
created_at, oneof op { AddMemberOp | RemoveMemberOp | UpdateMetadataOp
| CompactOp }, signature }, plus a `group_op` variant on
GroupContainer (regenerated).
- crdt_wire.rs: sign_op (sign the signature-empty encoding with the
actor's identity key) and verify_and_decode (resolve the actor pubkey
by PeerId — supplied by the caller — verify the signature, check
actor_id matches the key, validate op_id length, decode into a core
GroupOp). proto<->core conversion for all op kinds.
- placeholder receive arm for the GroupOp container variant (real
merge/apply handler lands in the integration slice).
5 unit tests: sign/verify round-trip, all op kinds, tampered op
rejected, wrong-key rejected, non-16-byte op_id rejected. Builds clean;
full libqaul suite passes (136).
Third slice of the group CRDT (docs/proposals/Group-State-CRDT.md): durable op storage and the capability advertisement, so received ops can accumulate and the view be rebuilt across restarts. - storage: GroupAccountDb gains a per-account `group_ops` sled tree (key = group_id ++ op_id), wired through all construction sites incl. the graceful tree-open fallbacks. - crdt_store.rs: save_op (persist an already-verified GroupOp), load_crdt (rebuild a GroupCrdt for a group by folding its stored ops; trusts stored ops, no re-verify), has_op. crdt_wire::decode_trusted decodes a stored op without signature re-check. - Capabilities::GROUP_CRDT = 1 << 3, OR'd into Capabilities::LOCAL so this binary advertises CRDT support; peers without the bit keep the legacy invite/reply path. - placeholder GroupOp receive arm already present from the prior slice. 3 store tests (save/load round-trip + derive, per-group isolation, add→remove sequence reloads converged). 139 libqaul tests pass; builds clean. Remaining (final, load-bearing slice — needs live verification): the receive handler (verify via router pubkey → save_op → re-derive view → reconcile into Group), outbound op production on add/remove/rename, founder tracking on Group, capability-gated translation to/from the legacy invite path, and retiring the Group::revision merge.
Fourth slice (docs/proposals/Group-State-CRDT.md): make the CRDT observable and exercised end-to-end, verifiable via qauld-ctl. - Group gains a `founder` field (serde default), set to the creator on group create, and threaded to invitees via GroupInfo.founder so they can derive the membership CRDT. - Receive: GroupContainer::GroupOp -> Group::on_crdt_op resolves the actor's public key (Users::get_pub_key), verifies the signature (crdt_wire::verify_and_decode), and persists the op (crdt_store). - Emit: Group::emit_crdt_op signs an op with the local identity key, persists it, and sends it (wrapped as a group notify so it routes to Group::net) to current members plus, for an Add, the new member. Hooked additively onto the existing invite (Add), remove (Remove, tombstoning observed adds) and rename (UpdateMetadata) handlers — the legacy path is unchanged. - Read: GroupCrdtView RPC + Group::crdt_view derive the converged view; qauld-ctl `group crdt-view --group-id <id>` prints founder, name, op_count and members/roles. Live-verified on two nodes via qauld-ctl: A creates a group, invites B, B accepts; A's AddMember op and a subsequent rename's UpdateMetadata op propagate so BOTH nodes show an identical derived view (founder=admin, B=member, name converged). 139 libqaul tests pass; builds clean. KNOWN CAVEAT (same first-contact gap as rotation/envelope): emit_crdt_op sends under the per-peer session, so an op produced before that session is established (e.g. the AddMember at the very first invite to a peer with no prior session) fails to encrypt and is dropped — warm the session first (any prior message) and ops flow. Proper fix is session-tolerant/queued delivery, shared with the rotation/envelope work. Also still pending: making the CRDT the source of truth for Group.members (today it shadows the legacy list) and retiring Group::revision.
Fifth slice (docs/proposals/Group-State-CRDT.md): the derived CRDT view now drives the materialized Group.members / name that every existing read path uses (chat routing, group info, UI), so CRDT membership changes are authoritative — not just visible via `group crdt-view`. Group::reconcile_group_from_crdt rebuilds Group.members from the derived view after every op (emit + receive): - members in the view are inserted/updated with the view's role (preserving existing per-member fields); - a member the CRDT has an Add op for but that is no longer in the view was tombstoned by a Remove → dropped from Group.members; - a member the CRDT is silent about (no add op) is preserved, so a mixed/legacy group is not damaged and a freshly-invited peer is not dropped before its add op arrives (transition-safe). Metadata name (LWW) is applied when set. No-op for pre-CRDT groups (no founder). Wired into emit_crdt_op and on_crdt_op. Live-verified on two nodes via qauld-ctl: after invite the legacy `group info` shows the CRDT membership (founder + B); after a CRDT remove of B the legacy `group info` drops to 1 member and crdt-view shows B tombstoned — i.e. the CRDT, not the rev-counter, now determines membership. 139 libqaul tests pass; builds clean. Note: the rev-counter (Group::revision) merge still exists for legacy GroupInfo gossip between pre-CRDT peers; it is now shadowed by CRDT reconciliation for CRDT-managed groups and can be retired once all peers advertise GROUP_CRDT. Session-tolerant op delivery (the first-contact cold-session gap) remains shared follow-up work.
Sixth slice (docs/proposals/Group-State-CRDT.md): the tombstone/epoch garbage-collection story, with a correctness fix to the core. Core fix: recompute_compaction previously pruned ALL ops below the compaction floor, which would drop live member adds / winning metadata set before the floor — silently changing the derived view. view() is refactored into derive() which also returns the load-bearing op_ids (live member adds + winning name/avatar writes); compaction now keeps those (and the Compact ops) while collapsing only dead history, so the view is invariant under compaction. Regression test compaction_preserves_live_view. Operator surface: - Group::crdt_compact emits a signed Compact op (epoch+1, floor = given `below` or the current max lamport); admin-only is enforced by the CRDT at apply time. GroupCrdtCompactRequest/Response RPC; epoch added to GroupCrdtViewResponse. - qauld-ctl `group crdt-compact --group-id <id> [--below N]`, and `group crdt-view` now prints the epoch. Live-verified via qauld-ctl: after 4 renames (4 ops) a compaction collapses to 2 ops (winning metadata + Compact), epoch 0->1, with the derived name and membership unchanged. 140 libqaul tests pass; builds clean.
Final slice: remove the rev-counter's authority over membership. In on_group_notify, a CRDT-managed group (one with a founder) now skips the legacy revision-gated GroupInfo merge entirely — its membership and metadata are owned by the CRDT (crdt.rs) and maintained by reconcile, so incoming legacy gossip can no longer clobber CRDT-derived state. The `revision` field is kept (not deleted) so backward-compatible GroupInfo gossip with pre-CRDT peers still works and stored groups still deserialize; it is simply no longer consulted for CRDT groups. Field doc updated to mark it legacy/retired. Live-verified via qauld-ctl: with the rev merge retired, a cold-start invite + rename still converge on B purely through the CRDT — the legacy `group info` read path shows 2 members and the new name, matching crdt-view. 140 libqaul tests pass; builds clean. This completes the group membership/metadata CRDT: core OR-Set + LWW, signed-op wire, persistence, capability gating, live integration as the source of truth, compaction GC, session-tolerant delivery, and now the retirement of the rev-counter merge. Remaining (optional, future): translate a legacy peer's GroupInfo changes into CRDT ops for genuinely mixed groups.
- with_group_mut() on main now takes a GroupSaveReason; the CRDT view sync picks Renamed when the LWW name changes (search reindex) and MembershipChanged otherwise. - Regenerate qaul.rpc.group.rs: crdt-view/compact oneof fields moved to tags 18-21 because group_search_request took 17 on main.
dastansam
force-pushed
the
feat/group-membership-crdt
branch
from
July 24, 2026 05:24
8f62a67 to
e6d11fd
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the
Group::revisionlast-writer-wins merge with a signed-opCRDT so partitioned members converge deterministically on membership and
metadata. Implements docs/proposals/Group-State-CRDT.md.
remove an admin. Gated on Capabilities::GROUP_CRDT.
and session-tolerant delivery (queues sends during first-contact
handshake instead of dropping). Inspect via
qauld-ctl group crdt-view.