Skip to content

fix: Redis cache, BullMQ webhook queue, pg_trgm migration, Prisma pool tuning (#775 #776 #777 #778) - #893

Merged
OlaGreat merged 3 commits into
OlaGreat:mainfrom
trinnode:fix/775-776-777-778-backend-improvements
Jul 25, 2026
Merged

fix: Redis cache, BullMQ webhook queue, pg_trgm migration, Prisma pool tuning (#775 #776 #777 #778)#893
OlaGreat merged 3 commits into
OlaGreat:mainfrom
trinnode:fix/775-776-777-778-backend-improvements

Conversation

@trinnode

Copy link
Copy Markdown
Contributor

Summary

This PR addresses four backend infrastructure issues in a single focused changeset:

#775 — Replace in-process leaderboard LRU cache with Redis

  • Added ioredis as a dependency
  • Created src/services/redis.ts — shared Redis client constructed from REDIS_URL env var with graceful fallback to in-process cache when Redis is unavailable (local dev still works out of the box)
  • Updated getCachedLeaderboard, setCachedLeaderboard, and invalidateProfileLeaderboardCache to use redis.get/redis.setex/redis.del when Redis is available, falling back to the existing in-process Map cache
  • Cache invalidation across processes is now possible when a new transaction arrives

#776 — Replace DB-polled webhook delivery with BullMQ job queue

  • Added bullmq as a dependency
  • Created src/services/webhook-queue.ts — BullMQ queue and worker with:
    • Built-in exponential backoff: { attempts: 4, backoff: { type: 'exponential', delay: 1000 } }
    • Concurrency control (5 concurrent workers, 10 jobs/second rate limit)
    • DB delivery log kept in sync via worker lifecycle hooks
  • Updated webhook-processor.ts to start a BullMQ worker when Redis is available, falling back to the existing DB-polled processor for local dev
  • Webhook deliveries are now enqueued immediately in BullMQ when created, eliminating poll-interval latency

#777 — Add pg_trgm extension creation to Prisma migration

  • The pg_trgm extension migration already exists at prisma/migrations/20260627000000_add_pg_trgm_extension/ — no additional changes needed

#778 — Expose Prisma connection pool size via environment variable

  • Updated src/db.ts to construct DATABASE_URL with connection_limit and pool_timeout search params from env vars
  • Added DATABASE_POOL_SIZE (default: 10, recommended: 20 for production) and DATABASE_POOL_TIMEOUT (default: 10 seconds) to .env.example
  • Requests now return 503 when the pool is exhausted instead of silently queuing

Additional fixes (pre-existing)

  • Fixed missing v1Router.get("/supporters/:address", ...) handler declaration that prevented compilation
  • Fixed webhook.secretwebhook.secretHash to match the Prisma schema field name
  • Fixed minor TS errors in test files that prevented clean builds

Test Results

  • Build passes cleanly (tsc -p tsconfig.json — zero errors, improved from 2 pre-existing syntax errors)
  • 99 tests pass, 13 fail (all pre-existing failures unrelated to this PR — same tests fail on the main branch)
  • Redis fallback verified: when REDIS_URL is not set, all features gracefully fall back to in-process cache and DB-polled processor

Dependencies Added

  • ioredis — Redis client
  • bullmq — job queue built on Redis

Closes #775
Closes #776
Closes #777
Closes #778

…xpose Prisma pool tuning (OlaGreat#775 OlaGreat#776 OlaGreat#777 OlaGreat#778)

- Add ioredis and bullmq as dependencies
- Create Redis client service (src/services/redis.ts) with graceful fallback
  to in-process cache when REDIS_URL is not set
- Replace in-process LRU leaderboard cache with Redis-backed cache that
  falls back to in-process cache for local dev (OlaGreat#775)
- Create BullMQ webhook delivery queue (src/services/webhook-queue.ts) that
  replaces DB-polled webhook processor when Redis is available, with
  exponential backoff and concurrency control (OlaGreat#776)
- Update webhook-processor to use BullMQ when Redis is available, falling
  back to existing DB-polled approach for local dev
- Expose DATABASE_POOL_SIZE and DATABASE_POOL_TIMEOUT env vars in db.ts
  to allow Prisma connection pool tuning (OlaGreat#778)
- pg_trgm extension migration already exists (OlaGreat#777)
- Fix pre-existing TS build errors: missing supporters route handler,
  webhook secret vs secretHash field mismatch
- Add REDIS_URL, DATABASE_POOL_SIZE, DATABASE_POOL_TIMEOUT to .env.example
@drips-wave

drips-wave Bot commented Jul 25, 2026

Copy link
Copy Markdown

@trinnode Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

…xpose Prisma pool tuning (OlaGreat#775 OlaGreat#776 OlaGreat#777 OlaGreat#778)

- Add ioredis and bullmq as dependencies
- Create Redis client service (src/services/redis.ts) with graceful fallback
  to in-process cache when REDIS_URL is not set
- Replace in-process LRU leaderboard cache with Redis-backed cache that
  falls back to in-process cache for local dev (OlaGreat#775)
- Create BullMQ webhook delivery queue (src/services/webhook-queue.ts) that
  replaces DB-polled webhook processor when Redis is available, with
  exponential backoff and concurrency control (OlaGreat#776)
- Update webhook-processor to use BullMQ when Redis is available, falling
  back to existing DB-polled approach for local dev
- Expose DATABASE_POOL_SIZE and DATABASE_POOL_TIMEOUT env vars in db.ts
  to allow Prisma connection pool tuning (OlaGreat#778)
- pg_trgm extension migration already exists (OlaGreat#777)
- Fix pre-existing TS build errors: missing supporters route handler,
  webhook secret vs secretHash field mismatch
- Add REDIS_URL, DATABASE_POOL_SIZE, DATABASE_POOL_TIMEOUT to .env.example
Adds missing migrations for pre-existing drift between schema.prisma
and the migration history:
- Rename Webhook.secret → secretHash
- Add unique constraint on AcceptedAsset(profileId, code, issuer)
- Add unique constraint on Profile(walletAddress)
- Recreate SupportTransaction FK with ON DELETE RESTRICT
@OlaGreat
OlaGreat merged commit c1478f6 into OlaGreat:main Jul 25, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants