Intermittent fail in Integration Tests for the cache_redis plugin because Redis cluster race condition

[swcurran]

Claude summary and actions. I'll copy/paste this into Claude Code to fix the issue in a separate PRs from weekly updates.

Summary

redis_events integration tests intermittently fail with test_deliverer_pulls_messages_from_queue_and_sends_them and test_relay_has_keys_in_recip_key_uid_map failing. Root cause is a startup race in redis_events/integration/docker-compose.yml, unrelated to any plugin code change.

Evidence

CI run: https://github.com/openwallet-foundation/acapy-plugins/actions/runs/30927035404/job/92051904487

• faber throws at startup:

  acapy_agent.transport.error.TransportError: No Redis instance setup, All slots are not covered after query all startup_nodes. 10922 of 16384 covered...
  

  This kills its InboundTransportManager/OutboundTransportManager startup tasks.
• test_deliverer_pulls_messages_from_queue_and_sends_them fails (assert 1 == 2) — a message pushed to acapy_outbound is never drained/delivered.
• test_relay_has_keys_in_recip_key_uid_map fails (assert {}) — relay never wrote to Redis.
• All 6 Redis Cluster nodes actually reach Cluster state changed: ok fine in the logs — the cluster forms correctly, just not before dependent services query it.
• Every other plugin's integration tests passed in the same run; this is isolated to redis_events.

Root cause

In redis_events/integration/docker-compose.yml:

• The redis-cluster service runs redis-cli --cluster create ... in its own container to form the 6-node cluster. Slot assignment + gossip propagation takes a couple of seconds after the nodes' ports open (observed: ports open at T+0s, Cluster state changed: ok on all nodes only by ~T+2s).
• relay, deliverer, and faber gate their startup via docker-compose-wait on WAIT_HOSTS=redis-node-3:6379 (plus a fixed WAIT_BEFORE of 5–10s). This only checks that one node's TCP port is open, not that the cluster has finished forming with full 16384-slot coverage.
• Nothing gates relay/deliverer/faber on the redis-cluster container's redis-cli --cluster create step actually completing (no depends_on: condition: service_completed_successfully).

Normally the fixed WAIT_BEFORE delay provides enough margin, but under CI resource contention the cluster-formation step can run slower than usual, and the race is lost intermittently.

redis-py (the redis package) version was unchanged in the triggering PR — only cryptography was bumped in redis_events/poetry.lock — so this is not a dependency-version regression.

Suggested fix direction

Make relay, deliverer, and faber actually wait for the cluster to be healthy rather than for one port to be open:

• Drop tail -f /dev/null from the redis-cluster service's command so it exits after redis-cli --cluster create succeeds, and change relay/deliverer/faber to depends_on: redis-cluster: condition: service_completed_successfully; or
• Add a script/healthcheck that polls redis-cli --cluster check <host>:<port> (or CLUSTER INFO for cluster_state:ok across all 6 nodes) before those services proceed.

File: redis_events/integration/docker-compose.yml