Skip to content

Repository files navigation

Fideicomiso X — WhatsApp Bot

Conversational assistant for the clients of Fideicomiso X. It lets clients check their account status, report payments by sending receipts over WhatsApp, and escalate queries to a human advisor. Receipts are validated automatically with AI and archived in Google Drive. The bot also sends proactive reminders for upcoming and overdue installments through a daily cron.

Note on language: the bot's user-facing responses are in Spanish, since its clients are Spanish-speaking. All of that response copy lives in app/core/templates.py and is intentionally left in Spanish — it is product copy, not code. This README (developer-facing documentation) is in English.

Portfolio note: this is a sanitized public fork of a bot originally built for a client. The client's real identity, credentials, and infrastructure addresses have been replaced with neutral placeholders (Fideicomiso X, example.com, RFC-reserved IPs, etc.). The architecture and code are intact.


Stack

  • Language: Python 3.12
  • Web framework: FastAPI (async) on Uvicorn
  • Database: PostgreSQL managed by Supabase (accessed via asyncpg)
  • Channel: WhatsApp Business Cloud API (Meta Graph API v25.0)
  • AI: OpenAI (GPT-4o-mini) — intent classification and receipt validation with vision
  • Storage: Google Drive API v3 (Service Account + Shared Drive)
  • PDF extraction: PyMuPDF
  • Logging: python-json-logger (structured logs)
  • Cron: host cron (/etc/cron.d/) on the production VM
  • Deploy: AWS Lightsail (Ubuntu 22.04) + Docker Compose + Caddy 2 (automatic Let's Encrypt TLS)

Exact dependencies in pyproject.toml.


Architecture

WhatsApp client                            Cron (host cron on Lightsail, Mon–Fri 12:00 UTC)
      │                                            │
      ▼                                            ▼
Meta Graph API ──► POST /webhook/whatsapp     POST /internal/notifications/run
                       │                               │
                       ▼                               ▼
              HMAC verification               X-Internal-Secret header
                       │                               │
                       ▼                               ▼
            ┌──────────┴──────────┐           Enqueue upcoming/overdue installments
            │ Text?  Doc?  Btn?   │              into outbox `notificaciones`
            ▼                     ▼                       │
    Fast-path keywords   Vision validation (OpenAI)       ▼
    + LLM classifier             │                Drain outbox →
    (intent + confidence)        ▼                send_whatsapp_template
            │              Upload to Google Drive         │
            ▼                     │                       ▼
    Conversational FSM            ▼                Reconcile via
    (app/core/conversation.py)  Insert into         Meta status webhook
            │              pagos_reportados
            ▼
    Response via Meta API

Flow summary:

  1. Meta sends every message to POST /webhook/whatsapp with an HMAC signature.
  2. The bot validates the signature, identifies the client by phone number, and enters a transactional lock on user_states to avoid race conditions.
  3. Based on the current state and the detected intent (digit shortcut, button reply, opt-out keyword, or LLM), it produces a response and/or advances the FSM.
  4. Documents are processed in the background: AI validation, upload to Drive, persistence in the DB.
  5. In parallel, a daily cron invokes /internal/notifications/run, which enqueues reminders and dispatches the pending ones via WhatsApp templates.

Project structure

app/
├── main.py                         # FastAPI app + lifespan + logging
├── config.py                       # Settings (Pydantic) from .env
├── db.py                           # asyncpg pool + per-phone_number lock
├── routers/
│   ├── whatsapp.py                 # WhatsApp webhook (GET verify + POST)
│   └── internal.py                 # Internal cron endpoints (X-Internal-Secret)
├── core/
│   ├── conversation.py             # FSM + intent resolution + opt-out
│   ├── notifications_policy.py     # Business days, holidays, offset parsing
│   └── templates.py                # Spanish response copy (all replies)
├── models/
│   ├── message.py                  # WhatsApp and internal message schemas
│   └── notifications.py            # Pydantic schemas for the /test endpoint
└── services/
    ├── intent_classifier.py        # LLM: free text → Intent enum
    ├── document_ai.py              # Vision: validates receipt + extracts data
    ├── drive.py                    # Upload to Google Drive
    ├── pagos_reportados.py         # Queries: clientes, cuotas, pagos, opt-out
    ├── notifications.py            # Outbox: enqueue, drain, reconcile
    ├── whatsapp_sender.py          # Message sending / interactive menu / templates
    ├── whatsapp_media.py           # Downloading files from WhatsApp
    ├── escalations.py              # Human-advisor escalation flow
    └── injection_guard.py          # Prompt injection detection
supabase/
├── config.toml                     # Supabase CLI config
└── migrations/                     # Versioned SQL schema
.github/workflows/
└── notifications-cron.yml          # Manual fallback (workflow_dispatch) — the real cron runs on the host
deploy/
├── cron-notifications              # File for /etc/cron.d/whatsapp-bot-notifications
└── RUNBOOK.md                      # Operations, deploy, secret rotation, domain migration
Dockerfile                          # Multi-stage (uv builder → slim runtime)
docker-compose.yml                  # app + caddy services
Caddyfile                           # Reverse proxy + automatic TLS

Quickstart (local development)

Requirements

  • Python 3.12 (see requires-python in pyproject.toml)
  • A Supabase account with a project created
  • A Meta for Developers app with WhatsApp Business configured
  • A Google account with a GCP project and OAuth credentials
  • An OpenAI API key with access to GPT-4o-mini

Steps

# 1. Create a virtual environment and install dependencies (recommended: uv)
uv sync

# 2. Create a .env file with the required variables (see the next section)

# 3. Apply the Supabase migrations to the remote project
npx supabase@latest login
npx supabase@latest link --project-ref <PROJECT_REF>
npx supabase@latest db push

# 4. Start the server
uv run fastapi dev

By default it listens on http://localhost:8000. To expose the webhook to Meta during development, using a tunnel (ngrok, cloudflared) is recommended.


Environment variables

All are read from .env in development. Defined in app/config.py.

Core

Variable Required Description
DATABASE_URL Yes URL of the Supabase connection pooler (port 6543, not the direct 5432)
WA_ACCESS_TOKEN Yes Permanent Meta Business token for sending messages
WA_PHONE_NUMBER_ID Yes ID of the WhatsApp Business number
WA_WEBHOOK_VERIFY_TOKEN Yes Arbitrary token configured in Meta to verify the webhook
WA_META_APP_SECRET Yes Meta app secret, used to validate HMAC signatures on each request
OPENAI_API_KEY Yes API key with access to GPT-4o-mini
GOOGLE_SERVICE_ACCOUNT_JSON Yes Service Account JSON (on a single line, wrapped in single quotes)
DRIVE_ROOT_FOLDER_ID Yes ID of the root folder in the Shared Drive where receipts are archived
LLM_MODEL No OpenAI model for intent classification and receipt validation (default gpt-4o-mini)
APP_NAME No App name used for logging
ENVIRONMENT No development / staging / production
LOG_LEVEL No INFO by default
BUSINESS_TIMEZONE No Business timezone (default America/Argentina/Buenos_Aires)

Notifications (cron)

Variable Required Description
NOTIFICATIONS_INTERNAL_SECRET Yes X-Internal-Secret header required by /internal/* (cron auth)
WA_UPCOMING_TEMPLATE_NAME Yes Name of the Meta-approved template for upcoming installments
WA_OVERDUE_TEMPLATE_NAME Yes Name of the Meta-approved template for overdue installments
WA_TEMPLATE_LOCALE Yes Template locale (e.g. es_AR)
NOTIFICATIONS_UPCOMING_DAYS No Advance-notice offsets in days, comma-separated (default 7,2)
NOTIFICATIONS_OVERDUE_DAYS No Post-due-date offsets for Phase 1 (default 1,3,7)
NOTIFICATIONS_OVERDUE_WEEKLY_DAY No Weekly day for Phase 2 (0=Monday, default 0)
NOTIFICATIONS_QUIET_DAYS No Days with no sending, comma-separated (default 5,6 — Saturday/Sunday)
NOTIFICATIONS_DAILY_CAP No Maximum sends per drain run (default 500)
NOTIFICATIONS_TEST_RECIPIENT No Phone number used by /internal/notifications/test when none is passed

Advisor escalation

Variable Required Description
ESCALATION_ADVISOR_PHONE No Phone number of the advisor who receives the escalation notification. Empty → the mailto fallback is used
WA_ESCALACION_ASESOR_TEMPLATE_NAME No Meta-approved template to notify the advisor (vars: nombre, telefono, motivo, resumen). Default hello_world → falls back to mailto until the real template is configured

Note: several variables have legacy aliases (e.g. WHATSAPP_TOKEN instead of WA_ACCESS_TOKEN). See app/config.py for the complete list.

A ready-to-fill production template lives in deploy/env.production.example (key names with empty values).


Database

Versioned schema in supabase/migrations/. Main tables:

Table Purpose
clientes Trust holders (name, phone, trust, activo, opt-out)
cuotas Monthly installments per client (amount, fecha_vencimiento, estado, periodo_label)
pagos_reportados Uploaded receipts (wamid, Drive URL, status_verificacion, monto_pagado)
user_states Conversational FSM state per phone number
notificaciones Notification outbox (pending → sending → sent/failed/delivered/read)
pago_aplicaciones Ledger of each payment → installment allocation (audit + trigger idempotency)
creditos_cliente Surplus from approved payments, applied manually by the agent
feriados Holidays for the year, used by the cron to shift sends to the previous business day

The verification states of a payment are: pendiente_revision, aprobado, rechazado, duplicado.

The states of a notification are: pending, sending, sent, delivered, read, failed, skipped, escalated. The sent → delivered → read transition is driven by Meta's status webhook via reconcile_status.

Applying approved payments (FIFO + credits)

When the bot persists a receipt in pagos_reportados, it stores in monto_pagado the amount the OCR (OpenAI vision) extracts from the PDF/image. pagos_reportados.cuota_id remains an informational pointer (traceability); application is NOT based on it, but on a FIFO walk over all of the client's installments.

The trigger trg_apply_pago_on_approval (AFTER INSERT OR DELETE OR UPDATE OF status_verificacion, monto_pagado) runs fn_apply_pago_on_approval, which acts as a dispatcher:

  • New approval (transition to aprobado on INSERT or UPDATE) → fn_aplicar_pago(pago_id, notify := true).
  • Reversal, amount correction, cliente_id change, or DELETE of a payment that was aprobadofn_replay_cliente(cliente_id) (full recalculation).

fn_aplicar_pago(p_pago_id, p_notify, p_tolerancia := 1) — applies an approved payment:

  1. Locks (FOR UPDATE) the client's unsettled installments (pendiente | parcial | vencido), ordered by fecha_vencimiento ASC, id ASC.
  2. Assigns monto_pagado to the remaining balance of each installment in that order and records each allocation in pago_aplicaciones. Progress accumulates in cuotas.monto_pagado_acumulado. Each touched installment becomes parcial, or pagado if the remaining shortfall is <= p_tolerancia ($1 by default).
    • ⚠️ The tolerance decouples state and ledger: an installment can end up pagado with monto_pagado_acumulado < monto (shortfall ≤ $1, receipt rounding). Queries that detect debt must filter by state (<> 'pagado'), not only by the ledger (monto > monto_pagado_acumulado), or phantom cents-of-debt would reappear.
  3. Any surplus (> $0.005), once all installments are settled, is inserted into creditos_cliente with applied_at = NULL. Credits are not auto-consumed: applying them is a manual action by the agent (see deploy/RUNBOOK.md).
  4. If p_notify (only on new approvals, not on replay), it enqueues a pago_aprobado notification. It is idempotent: if the payment already has rows in pago_aplicaciones, it returns without doing anything.

fn_replay_cliente(p_cliente_id) — recalculates from scratch: resets the client's installments (monto_pagado_acumulado = 0, estado = 'pendiente'), purges its pago_aplicaciones and the auto-generated creditos_cliente (preserving the manual ones, with source_pago_id IS NULL), and reapplies all of its aprobado payments in created_at ASC order with notify := false. It does not touch notificaciones.

⚠️ The reset forces estado = 'pendiente', assuming every unsettled installment starts that way. If in the future something marks an installment as vencido or en_disputa, a replay of that client would silently revert it to 'pendiente'. Today it is harmless (no path writes those states), but keep it in mind before introducing automatic disputes/overdue transitions.

If monto_pagado is NULL or <= 0 (OCR failed), fn_aplicar_pago is a no-op. Implementation of the three functions and the pago_aplicaciones table: supabase/migrations/20260601000000_pago_aplicaciones_ledger_and_replay.sql.

Applying schema changes

# Create a new migration
npx supabase@latest migration new <name>

# Apply to the remote
npx supabase@latest db push

⚠️ The repo is NOT the source of truth for the live schema. Historically several changes were applied by hand (Supabase SQL editor), so the history of supabase_migrations.schema_migrations in the database does not match the files in supabase/migrations/ — missing, among others, the migrations for the v_estado_de_cuenta view. Before a db push or a db reset against the production database, check for drift; rebuilding from the repo does not reproduce the current state. This repo's migrations are written to be idempotent (CREATE OR REPLACE, IF NOT EXISTS) precisely so that documenting the live state without re-applying it is safe.

The backend connects with the Supabase connection pooler credential (see DATABASE_URL). Table access bypasses RLS by convention — it is a trusted backend.


HTTP endpoints

Method Path Auth Description
GET/HEAD /health Public Health check ({"status": "ok"})
GET /webhook/whatsapp Verify token Webhook verification by Meta (challenge/response)
POST /webhook/whatsapp HMAC Ingestion of WhatsApp messages and events
POST /internal/notifications/run X-Internal-Secret Enqueue + drain of the notifications outbox
POST /internal/notifications/test X-Internal-Secret Trigger a template to a specific number (debugging)

Notifications cron

The cron runs on the Lightsail VM via /etc/cron.d/whatsapp-bot-notifications (template file in deploy/cron-notifications), Monday to Friday at 12:00 UTC (09:00 ART):

0 12 * * 1-5 ubuntu curl -fsS -X POST http://localhost:8000/internal/notifications/run \
    -H "X-Internal-Secret: $(cat /opt/whatsapp-bot/.cron-secret)" \
    >> /var/log/whatsapp-bot-cron.log 2>&1

The secret is read from /opt/whatsapp-bot/.cron-secret (chmod 600) so it is not exposed in ps. Logs are rotated via /etc/logrotate.d/whatsapp-bot (14 days, compress).

The workflow .github/workflows/notifications-cron.yml is left disabled (no schedule:) and is kept only as a manual fallback (workflow_dispatch) in case the VM is down. Secrets required in the repo for that fallback: APP_URL and NOTIFICATIONS_INTERNAL_SECRET.

The endpoint response includes:

  • enqueued_upcoming / enqueued_overdue / enqueued_overdue_weekly — how many notifications were added to the outbox.
  • drained — how many were sent / failed / skipped in this run.
  • opted_out_with_pending — how many active clients with pending installments are silenced (compliance receipt).
  • failures_today — list of the day's failed entries with client, kind, and Meta error — useful for diagnosis (codes 190, 131026, etc.).

Conversational flow

Defined in app/core/conversation.py. FSM states:

  • GREETING → first contact; shows the interactive menu and moves to AWAIT_INTENT.
  • AWAIT_INTENT → waits for user input; resolves intent via digit shortcut, button reply, keyword fast-path, or LLM.
  • AWAIT_INVOICE_UPLOAD → waits for a file (PDF/image) of the receipt.

Intent resolution

The priority order for classifying user text:

  1. Digit shortcut (1, 2, 3) — from the legacy menu.
  2. Button reply (from the interactive menu buttons or templates).
  3. Keyword fast-path — opt-out (BAJA, PARAR, STOP, CANCELAR, NO MOLESTAR, DEJAR DE RECIBIR) and opt-in (REACTIVAR, VOLVER, SUSCRIBIR, START).
  4. LLM (GPT-4o-mini) — free-text classification with a confidence threshold.

Recognized intents (see app/services/intent_classifier.py):

  • STATUS — check account status (runs an immediate query).
  • AWAIT_INVOICE — report a payment (transitions to AWAIT_INVOICE_UPLOAD, after a duplicate check).
  • ESCALATE — talk to a human advisor.
  • OPT_OUT / OPT_IN — silence/reactivate automatic notifications (compliance with WhatsApp Business Policy).
  • GREETING / UNKNOWN — fall back to the menu.

Interactive menu

The main menu uses WhatsApp interactive buttons with three options: "Estado de tu cuenta", "Subir un comprobante", "Hablar con un asesor". Internally it is routed via a sentinel (<<INTERACTIVE_MENU>>) that the router detects and replaces with the sending of an interactive-type payload. The buttons return button_reply.id (e.g. STATUS), which maps directly to the corresponding intent.

Duplicate check

Before transitioning to AWAIT_INVOICE_UPLOAD, the bot checks whether the client already has a payment in pendiente_revision for the current month's installment. If so, it informs the client and blocks the send. Additionally, at the webhook level, the wamid is used for idempotency (Meta may retry deliveries).

Opt-out

When a client sends an opt-out keyword, clientes.notifications_opted_out = TRUE is set with notifications_opted_out_at = NOW(). The client stays active (activo = TRUE) — they can start conversations, check status, and upload receipts — but are excluded from the cron enqueues. To receive notifications again, they simply send an opt-in keyword.


Notifications (daily cron)

Implementation: app/services/notifications.py. Pattern: transactional outbox.

Phases

  • Phase 1 — Upcoming: advance notices before the due date (configurable offsets, default T-7 and T-2 business days).
  • Phase 1 — Overdue: post-due-date notices, anchored to the oldest pending installment (default T+1, T+3, T+7).
  • Phase 2 — Weekly: once the Phase 1 offsets are exhausted, a weekly reminder on the configured day (default Monday).
  • Approved payment: when an agent changes pagos_reportados.status_verificacion to aprobado, a DB trigger enqueues a kind='pago_aprobado' notification with a JSONB snapshot of the outcome (settled installments, surplus to creditos_cliente). It is delivered via the WA_PAGO_APROBADO_TEMPLATE_NAME template (example_pago_aprobado_v1) on the next cron run. Idempotent: a single pago_aprobado per pagos_reportados.id.

Guarantees

  • Daily idempotency via the ON CONFLICT DO NOTHING constraint on (cliente_id, cuota_id, kind, scheduled_for).
  • Safe concurrency via FOR UPDATE SKIP LOCKED when claiming the batch.
  • Retry with cap (MAX_ATTEMPTS=3); after that the row ends up failed.
  • Sweep of stale sending (>10 min) to recover from mid-drain crashes.
  • Bidirectional reconciliation: Meta's status webhook advances sent → delivered → read or marks failed, with anti-downgrade ranking.
  • Business days + holidays: if the target falls on a "quiet" day (default Sat/Sun) or on a feriados entry, it is shifted to the previous business day with previous_business_day.
  • Filter for payments in review: installments with pagos_reportados in state pendiente_revision or aprobado are excluded from the candidate set (so it doesn't bother someone who already paid).
  • Opt-out filter: clients with notifications_opted_out = TRUE are left out of the enqueue.

Run observability

Each cron run returns a JSON with counts per kind, a list of the day's failures, and a count of opted-out clients with pending installments. Useful for diagnosing problems without querying the DB.


External integrations

WhatsApp Business API (Meta)

  1. Create an app on Meta for Developers.
  2. Add the WhatsApp product and configure a number.
  3. In Configuration → Webhooks, set:
    • Callback URL: https://<your-domain>/webhook/whatsapp
    • Verify token: the value of WA_WEBHOOK_VERIFY_TOKEN
  4. Subscribe to the messages and message_template_status_update fields (the latter optional for reconcile).
  5. Generate a permanent token (System User) and load it into WA_ACCESS_TOKEN. Temporary tokens expire after 24h.
  6. Copy the App Secret to WA_META_APP_SECRET.
  7. Create and approve the notification templates in Meta Business Manager (the names go in WA_UPCOMING_TEMPLATE_NAME, WA_OVERDUE_TEMPLATE_NAME, WA_UPCOMING_SALDO_REMANENTE_TEMPLATE_NAME, WA_OVERDUE_INTERES_AVISO_TEMPLATE_NAME, and WA_PAGO_APROBADO_TEMPLATE_NAME). The reminder templates include 3 Quick Reply buttons with IDs STATUS, AWAIT_INVOICE, ESCALATE.

Google Drive

The bot uses a Service Account from the client's Google Workspace, operating on a Shared Drive of which the SA is a member. The files remain owned by the client's Workspace, not by a personal account. Setup flow:

  1. In Google Cloud Console (the client's project), enable the Google Drive API.
  2. Create a Service Account and download the JSON key.
  3. In Google Drive, create (or use) a Shared Drive and add the SA email as a member with "Content manager" permission.
  4. Copy the ID of the Shared Drive (or of a folder within it) to DRIVE_ROOT_FOLDER_ID.
  5. Paste the JSON contents into GOOGLE_SERVICE_ACCOUNT_JSON on a single line, wrapped in single quotes.

API calls are made with supportsAllDrives=True / includeItemsFromAllDrives=True to operate on Shared Drives.

The folder structure created in Drive is: {Year}/{Month}/{fideicomiso_id}_{cliente_nombre}/{date}.pdf.

OpenAI

GPT-4o-mini is used for two functions:

  1. Intent classification (app/services/intent_classifier.py) — free user text → intent enum with a confidence score.
  2. Receipt validation (app/services/document_ai.py) — analyzes the PDF/image, determines whether it is a valid bank receipt, and extracts amount, date, sender, receiver, and transaction ID.

Supabase

Only PostgreSQL is used (no auth, no storage). Connect with the connection pooler (port 6543) from serverless/edge backends.


Security

  • HMAC signature (X-Hub-Signature-256) validated on every webhook request using WA_META_APP_SECRET.
  • Gating by active client — the bot resolves the incoming number against clientes and rejects if it does not exist or activo = FALSE.
  • Protected internal endpoints via the X-Internal-Secret header.
  • Prompt injection guard (app/services/injection_guard.py) — heuristic check before the LLM.
  • Per-client lock on user_states (SELECT ... FOR UPDATE) — guarantees FSM atomicity against concurrent messages from the same number.
  • Idempotency by wamid in pagos_reportados — prevents duplicates on Meta retries.
  • Structured JSON logs — events serialized with internal IDs for traceability.

Deploy

Production runs on AWS Lightsail (Ubuntu 22.04, $10/mo plan: 1 GB RAM, 2 vCPU burst, 40 GB SSD) in us-east-1, with an attached static IP and <ip>.nip.io as the webhook FQDN (no dedicated domain registered, a deliberate decision at project close — see deploy/RUNBOOK.md for the procedure to migrate to a dedicated domain).

The runtime is packaged as Docker Compose (2 services):

  • app — multi-stage image built from the Dockerfile (python:3.12-slim + uv + uvicorn). Binds only to 127.0.0.1:8000 so the host cron can call it directly, without going through TLS.
  • caddycaddy:2-alpine, exposes :80 and :443, issues and renews Let's Encrypt certs automatically (HTTP-01 challenge). Config in Caddyfile.

Deploying changes:

ssh ubuntu@<static-ip>
cd /opt/whatsapp-bot
git pull
docker compose up -d --build

Hardening applied on the VM: unattended-upgrades (automatic security patches + reboot at 03:00), logrotate for the cron log. No automatic snapshots, by project decision.

Full operations, secret rotation, snapshot restore, migration to a dedicated domain, and manual payment approval: see deploy/RUNBOOK.md.


Next steps (roadmap)

  • Annual holiday seed — a process to load holidays for future years (currently only 2026).
  • Migration to a dedicated domain — optional, once a domain is registered. Procedure documented in deploy/RUNBOOK.md.
  • Admin dashboard — currently payment approval and creditos_cliente application are done with direct SQL.

About

FastAPI service integrating WhatsApp Business Cloud API, OpenAI, and Google Drive. Webhook with HMAC validation returns fast and defers processing to background tasks; internal cron endpoints are secret-authenticated; notifications use an outbox with independent retries and delivery reconciliation via Meta's status webhook.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages