A distributed order processing system built with Go, RabbitMQ, and PostgreSQL, demonstrating a choreography-based Saga pattern with full resilience guarantees.
When a client places an order, no service calls another service directly. Instead, each service reacts to events on RabbitMQ and publishes its own result. This is the Choreography-based Saga pattern.
When you run make docker-up, Docker starts 8 containers:
rabbitmq ← message broker (the "bus")
order-db ← PostgreSQL exclusive to order-service
payment-db ← PostgreSQL exclusive to payment-service
inventory-db ← PostgreSQL exclusive to inventory-service
shipping-db ← PostgreSQL exclusive to shipping-service
order-service ← the only HTTP-facing service
payment-service ← no HTTP, pure event consumer
inventory-service ← no HTTP, pure event consumer
shipping-service ← no HTTP, pure event consumer
Each service waits for its DB and RabbitMQ to be healthy before starting (depends_on with condition: service_healthy).
Each service does this on boot, in order:
1. Connect to its own PostgreSQL
2. Run embedded SQL migrations (go:embed — SQL files compiled into the binary)
3. Connect to RabbitMQ with exponential backoff retry
4. Declare exchanges, queues and DLQ bindings (idempotent)
5. Start goroutine consumers (one goroutine per queue)
6. [order-service only] Start HTTP server
Client
│
│ POST /orders
▼
order-service
├─ validates request
├─ INSERT into orders (status=PENDING) + order_items ← DB transaction
├─ publishes → order.created to order.exchange
└─ returns 201 { order_id }
RabbitMQ routes order.created to the queue payment.order-created.
payment-service (goroutine consuming payment.order-created)
├─ unmarshals payload
├─ checks idempotency_keys — skip if already seen
├─ simulates gateway call → APPROVED
├─ INSERT into payments (status=APPROVED) ← DB transaction
├─ publishes → payment.approved to payment.exchange
└─ ACKs the message
RabbitMQ fans payment.approved to two queues: inventory.payment-approved AND order.payment-approved.
inventory-service (goroutine consuming inventory.payment-approved)
├─ opens DB transaction
├─ SELECT … FOR UPDATE on all product rows ← locks prevent concurrent over-reservation
├─ checks available qty (total - reserved) for every item
├─ UPDATE quantity_reserved += qty for each item
├─ INSERT reservation + reservation_items
├─ publishes → inventory.reserved to inventory.exchange
└─ ACKs
order-service (goroutine consuming order.payment-approved)
└─ UPDATE orders SET status = 'PAYMENT_APPROVED'
RabbitMQ fans inventory.reserved to two queues: shipping.inventory-reserved AND order.inventory-reserved.
shipping-service (goroutine consuming shipping.inventory-reserved)
├─ calculates freight cost (base + items * 1.50)
├─ generates tracking code TRK-xxxxxxxx
├─ sets estimated delivery (3–7 days)
├─ INSERT into shipments
├─ publishes → shipping.scheduled to shipping.exchange
└─ ACKs
order-service (goroutine consuming order.inventory-reserved)
└─ UPDATE orders SET status = 'INVENTORY_RESERVED'
order-service (goroutine consuming order.shipping-scheduled)
└─ UPDATE orders SET status = 'CONFIRMED' ← happy path complete
If inventory runs out of stock:
inventory-service
├─ detects ErrInsufficientStock
├─ publishes → inventory.failed
└─ ACKs (graceful failure, no retry needed)
payment-service (consuming payment.inventory-failed)
├─ reverses / voids the payment
├─ UPDATE payments SET status = 'REVERSED'
└─ publishes → payment.reversed
order-service (consuming order.payment-reversed)
└─ UPDATE orders SET status = 'CANCELLED'
Manual ACK — pkg/messaging/consumer.go
The consumer only calls msg.Ack() after the handler succeeds. If the service crashes mid-processing, the message stays in the queue and gets redelivered.
Dead Letter Queues — services/*/internal/messaging/topology.go
Every queue is paired with a .dlq queue. If a handler fails 3 times, the message is nacked to the DLX and lands in the permanent DLQ for manual inspection — it never disappears.
Publisher Confirms — pkg/messaging/publisher.go
Publish() blocks until the broker sends an ACK. If RabbitMQ crashes between the DB write and the publish, the caller gets an error and can retry.
Idempotency keys — every service's repository
Every event carries a UUID event_id. Before processing, the service does:
INSERT INTO idempotency_keys (event_id) VALUES ($1) ON CONFLICT DO NOTHINGIf the same event arrives twice (redelivery after a crash), the second execution is a no-op — no double charges, no double reservations.
SELECT FOR UPDATE — services/inventory/internal/repository/inventory_repo.go
When two orders arrive simultaneously for the same product, both goroutines try to lock the same inventory rows. One wins, the other blocks until the first commits. This prevents selling stock you don't have.
Goroutines + graceful shutdown — every cmd/main.go
Each queue consumer is a goroutine. When SIGTERM arrives, cancel() stops accepting new messages and wg.Wait() lets in-flight handlers finish before the process exits.
| Routing Key | Publisher | Consumers |
|---|---|---|
order.created |
order | payment |
payment.approved |
payment | inventory, order |
payment.failed |
payment | order |
payment.reversed |
payment | order |
inventory.reserved |
inventory | shipping, order |
inventory.failed |
inventory | payment, order |
inventory.released |
inventory | payment |
shipping.scheduled |
shipping | order |
shipping.failed |
shipping | inventory, order |
PENDING
→ PAYMENT_APPROVED → INVENTORY_RESERVED → CONFIRMED (happy path)
→ PAYMENT_FAILED (payment declined)
→ PAYMENT_REVERSED (inventory or shipping failed, payment refunded)
→ INVENTORY_FAILED (out of stock, before reversal)
→ SHIPPING_FAILED (carrier error, before release)
→ CANCELLED (terminal failure, fully compensated)
tech-challenge/
├── go.mod / go.sum
├── Makefile
├── docker-compose.yml
├── pkg/
│ ├── events/events.go # event name constants + payload structs
│ ├── messaging/
│ │ ├── connection.go # AMQP connection with exponential backoff
│ │ ├── publisher.go # publish with broker confirms
│ │ └── consumer.go # goroutine consumer with manual ack + DLQ logic
│ └── database/postgres.go # pgxpool + fs.FS migration runner
└── services/
├── order/
│ ├── Dockerfile
│ ├── cmd/main.go
│ └── internal/
│ ├── config/
│ ├── domain/
│ ├── repository/ # pgx CRUD + idempotency
│ ├── messaging/ # topology, publisher, consumers (7 queues)
│ ├── http/ # gorilla/mux router, POST /orders, GET /orders/{id}
│ └── migrations/ # embedded SQL via go:embed
├── payment/ # same structure, no HTTP
├── inventory/ # same structure, SELECT FOR UPDATE
└── shipping/ # same structure, freight calculation
| Layer | Technology |
|---|---|
| Language | Go 1.22 |
| Message Broker | RabbitMQ 3.13 (AMQP 0-9-1) |
| Database | PostgreSQL 16 (one per service) |
| AMQP client | github.com/rabbitmq/amqp091-go |
| PostgreSQL driver | github.com/jackc/pgx/v5 |
| HTTP router | github.com/gorilla/mux |
| Container runtime | Docker + Docker Compose |
# Start everything
make docker-up
# Place an order
curl -s -X POST http://localhost:3001/orders \
-H "Content-Type: application/json" \
-d '{
"customer_id": "123e4567-e89b-12d3-a456-426614174000",
"payment_method": "credit_card",
"items": [
{
"product_id": "00000000-0000-0000-0000-000000000001",
"quantity": 2,
"unit_price": 49.90
}
]
}'
# Check order status
curl -s http://localhost:3001/orders/<order_id>
# Monitor queues
open http://localhost:15672 # RabbitMQ Management UI (admin / admin)
# View logs
make docker-logs
# Tear down (removes all volumes)
make docker-down