Cycle Data Pipeline

Automated pipeline for exporting IMPACT cycle data from MongoDB into per-cycle CSV bundles: combined site data, NDA data dictionaries, and patient statistics. Optionally uploads the cycle directory to S3.

Each cycle corresponds to a MongoDB database (e.g. 2026-JAN-data, 2026-JUL-data). You must run the pipeline against the database for that cycle — the database holds all collections (data structures) and documents for the batch you are exporting.

Prerequisites

• Go 1.18+
• MongoDB running and reachable (default: mongodb://localhost:27017)
• AWS CLI configured (only for S3 upload)
• Network access to the NDA data dictionary API

Setup

chmod +x setup.sh run_cycle.sh
./setup.sh

This installs Go dependencies and builds cycle_pipeline.

Running a Cycle

1. Ensure MongoDB has the cycle database loaded

The pipeline reads from a single MongoDB database per run. Database names follow the pattern YYYY-MON-data (e.g. 2026-JAN-data for a January 2026 batch). Confirm the database is available before starting:

docker ps | grep mongo   # if MongoDB runs in Docker

2. Run the pipeline

./run_cycle.sh --cycle 3 --month jul --year 2026

This creates output under cycle3_jul2026/. When prompted, select the matching MongoDB database (e.g. 2026-JUL-data).

To skip the database picker:

./run_cycle.sh --cycle 3 --month jul --year 2026 --db 2026-JUL-data

Or invoke the Go binary directly (no S3 prompt):

./cycle_pipeline --cycle 3 --month jul --year 2026 --db 2026-JUL-data

3. Review output and optionally upload to S3

The pipeline prints a summary when finished. run_cycle.sh then asks whether to sync the cycle folder to S3. To upload without prompting:

./run_cycle.sh --cycle 3 --month jul --year 2026 --db 2026-JUL-data --yes-s3

For manual S3 sync and additional options, see PIPELINE.md.

What the Pipeline Does

1. Infrastructure — Creates the cycle directory and subfolders (data/, data_dicts/, stats/).
2. Database selection — Connects to MongoDB (read-only) and prompts for the cycle database, unless --db is provided.
3. Data gathering — For each collection in that database, writes a combined-site CSV to data/ using site_mappings.csv for site assignment.
4. Patient stats — Writes count CSVs to stats/ (unique patients per structure, per site, and cycle-wide).
5. Data dictionaries — Fetches or copies NDA dictionary files into data_dicts/ (reuses a shared cache at repo-root data_dicts/).
6. Summary — Reports CSVs created, skipped, errors, and patient totals.

Output Structure

cycle3_jul2026/
├── data/                      # one CSV per data structure (all sites combined)
│   ├── aces01.csv
│   └── ...
├── data_dicts/                # NDA dictionary CSVs for this cycle
│   ├── aces01_dict.csv
│   └── ...
└── stats/                     # patient and structure counts
    ├── cycle_summary.csv
    ├── patients_per_structure.csv
    └── patients_per_structure_per_site.csv

data_dicts/                    # shared dictionary cache (repo root, across cycles)
└── ...

S3 upload mirrors the cycle directory:

s3://spinup-003034-impact-mh-data/cycle3_jul2026/
├── data/
├── data_dicts/
└── stats/

Common Options

Flag	Description
--cycle N	Cycle number (required)
--month jan|jul	Half-year token (required)
--year YYYY	Four-digit year (required)
--db NAME	MongoDB database for this cycle (e.g. 2026-JUL-data)
--mongo-uri URI	MongoDB connection (default: mongodb://localhost:27017)
--force-dicts	Re-fetch dictionaries from NDA API
--skip-dicts	Skip dictionary phase
--yes-s3	Upload to S3 without prompting (run_cycle.sh only)

Environment variable S3_BUCKET overrides the default bucket (spinup-003034-impact-mh-data).

Further Documentation

See PIPELINE.md for site mapping rules, ignored Mongo fields, patient stat definitions, manual S3 sync, idempotent reruns, and troubleshooting.