Race condition fixes: remaining work (Phase 5 + tests)

[mschilling]

Context

Phases 1-4 of the race condition fix plan have been implemented (follows up on PR #283 which fixed duplicate level-up events). The remaining work covers Phase 5 (workflow atomicity) and test coverage for all idempotency changes.

Completed

• Phase 1: Duplicate check moved before game action creation, deterministic IDs, idempotent GameActionService
• Phase 2: Async operations removed from Firestore transactions in RewardService (two-phase approach)
• Phase 3: Pub/Sub message idempotency tracking with _processedMessages collection + scheduled cleanup
• Phase 4: Badge activity IDs changed from Date.now() to deterministic format

Remaining Work

Phase 5: Workflow Atomicity (lower priority)

Investigate whether libs/server/progression-engine/src/lib/workflows/ exists or if multi-step workflows leave partial state on failure. If so, implement saga pattern or transaction batches for all-or-nothing state transitions.

Firestore TTL Policy

Configure a Firestore TTL policy on the _processedMessages collection's expiresAt field via the Firebase console. This supplements the scheduled cleanup function (cleanupProcessedMessages) that already runs every 6 hours.

Test Coverage

Unit Tests

• Test idempotent game action creation (call createIdempotent twice with same eventId, expect no error and same data returned)
• Test GameActionService.generateDeterministicId produces consistent IDs
• Test RewardService.grantReward two-phase approach (transaction commits state, side effects run after)
• Test RewardActivityService.saveActivityIdempotent returns false on duplicate
• Test RewardActivityService.generateBadgeActivityId for each trigger type
• Test Firestore transaction retry behavior under contention

Integration Tests

• Simulate concurrent webhook deliveries for the same event — verify only one game action is created
• Simulate Pub/Sub message redelivery — verify tryMarkMessageAsProcessed prevents duplicate processing
• Verify reward grant + badge grant + notification complete without data races

Load Tests

• 10+ concurrent requests for the same user — verify no duplicate activities
• Verify XP and counter accuracy after concurrent updates

Files Reference

File	Change
apps/github-receiver/src/app.ts	Reordered duplicate check
libs/server/integrations/src/lib/services/game-action.service.ts	Idempotent create, deterministic IDs
libs/server/progression-engine/src/lib/rewards/services/reward.service.ts	Two-phase transaction
libs/server/progression-engine/src/lib/rewards/services/reward-activity.service.ts	Deterministic badge activity IDs
apps/game-engine/src/triggers/progression-events.trigger.ts	Pub/Sub idempotency
apps/game-engine/src/triggers/cleanup.trigger.ts	Scheduled TTL cleanup