feat: add pg-write subcommand to bulk-write ASN info into PostgreSQL

[digizeph]

Adds an asninfo pg-write subcommand that bulk-writes the full ASN-info dataset into PostgreSQL via streaming COPY with an atomic staging-table swap, so API readers never see an empty table.

What it does

• Loads the same bgpkit-commons source build as generate (ASN names, as2org, population, hegemony, PeeringDB, countries), so field parity is by construction.
• Streams all rows into asninfo.current_staging via CSV-format COPY, builds indexes on staging (PK on asn, country btree, trigram GIN on name/org_name when pg_trgm is installed), then atomically swaps it over asninfo.current in one transaction (DROP current + RENAME), carrying index/constraint names to canonical names.
• When pg_trgm is not installed, the trigram indexes are skipped with a warning instead of failing.
• Records every run in asninfo.ingest_run (success and failure rows): task, status, row count, data_as_of, source_revision, timing, and error message.

Row shape (asninfo.current)

• Typed columns: asn bigint PK, name, country, country_name, org_id, org_name
• Per-source JSONB: population, hegemony, peeringdb (SQL NULL when a source has no data for the ASN)
• Provenance: data_as_of, source_revision

Usage

DATABASE_URL=postgresql://... asninfo pg-write

--database-url is accepted too; the env var is preferred (keeps credentials out of process listings).

Notes

• Connection: tokio-postgres 0.7.18 (fixes RUSTSEC-2026-0178).
• The dataset load runs on tokio's blocking pool: the blocking HTTP clients used by bgpkit-commons own nested tokio runtimes, and dropping such a runtime inside an async context panics on tokio >= 1.48. Note this same nested-runtime panic currently affects the pre-existing generate/serve data loads on main; this PR applies the blocking-pool fix to the new pg-write path only, fixing generate/serve would be a separate change.
• Also fixes three pre-existing clippy warnings in main.rs so the crate passes cargo clippy --all-targets -- -D warnings.
• Smoke-tested against a scratch PostgreSQL 17 database: 122,372 rows, two consecutive full-refresh runs, with and without pg_trgm, verifying the swap, index renames, provenance rows, and the no-pg_trgm degrade path.

[Copilot]

🟡 Changes recommended

Critical concurrency, TLS, and empty-dataset safeguards remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a pg-write command for bulk ASN ingestion into PostgreSQL using streaming COPY and staging-table replacement.

Changes:

• Adds PostgreSQL ingestion, indexing, and provenance tracking.
• Registers and documents the new command.
• Adds serialization tests and PostgreSQL dependencies.

Review findings:

• Critical (2 votes), src/pg_write.rs:151: Concurrent runs can interfere through the shared staging table. Hold a session-level advisory lock for the complete load.
• Moderate (2 votes), src/pg_write.rs:249: Replacing current loses grants and fails with dependent objects. Preserve the live relation or define an explicit grant/dependency strategy.
• Critical (2 votes), src/pg_write.rs:111: NoTls breaks TLS-required URLs and can expose traffic. Use TLS that honors sslmode.
• Moderate (1 vote), src/pg_write.rs:359: Failure records can contain fabricated provenance and row counts. Store NULL or actual progress.
• Nit (2 votes), src/pg_write.rs:245: Add lifecycle integration tests covering refreshes, rollback, indexes, and provenance.
• Moderate (2 votes), src/pg_write.rs:230: Schema-qualify gin_trgm_ops using the extension namespace.
• Critical (1 vote), src/pg_write.rs:198: Reject unexpectedly empty datasets before replacing production data.

File summaries

File	Description
src/pg_write.rs	Implements PostgreSQL ingestion, staging-table publication, indexing, provenance, and serialization tests.
src/main.rs	Registers and dispatches pg-write.
README.md	Documents PostgreSQL usage and schema.
CHANGELOG.md	Records the new feature.
Cargo.toml	Adds PostgreSQL and streaming dependencies.
Cargo.lock	Locks the updated dependency graph.

Review details

• Files reviewed: 5/6 changed files
• Comments generated: 7
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[digizeph]

Thanks for the review. Responses per finding:

1. Concurrent runs (advisory lock) — accepted. The loader now takes a session-level pg_advisory_lock right after connecting and holds it for the whole load, so overlapping runs serialize on the shared staging table. Verified live: a competing session holding the lock queues a second run, which proceeds immediately after release.
2. Grants lost on swap / dependent objects — real contract, now documented in the README. The swap intentionally replaces the table, so consumers must rely on default privileges (ALTER DEFAULT PRIVILEGES FOR ROLE asninfo_loader IN SCHEMA asninfo GRANT SELECT ON TABLES TO <consumer>); views/FKs depending on the table block the swap loudly, which is the intended failure mode.
3. TLS (NoTls) — declined. The loader is designed for a local/trusted connection; pointing it at a TLS-required server fails the connection safely rather than downgrading. Supporting sslmode would add a TLS stack, which is out of scope for this change.
4. Fabricated failure provenance — accepted. Error rows now record row_count = NULL and data_as_of = NULL; only successful swaps carry counts.
5. Lifecycle integration tests — declined for CI (this repo has no PostgreSQL service in CI; adding one is a separate decision). Lifecycle behavior was exercised against a local PostgreSQL: two consecutive full refreshes, advisory-lock blocking against a competing session, swap + index renames, pg_trgm present/absent paths, and provenance rows.
6. Schema-qualify gin_trgm_ops — accepted. The loader resolves the operator class in the extension's actual namespace (from pg_extension), so non-public installs work.
7. Empty-dataset guard — accepted. The loader refuses the swap when the loaded dataset is empty, or has fewer than half the rows currently loaded; the check is a pure function with unit tests.

[digizeph]

Superseded by direction change (2026-09-08): instead of extending the asninfo crate, the ASN/PeeringDB/IRR data load into PostgreSQL moves to a new standalone inserter with per-source schemas. The validated PG mechanics (advisory lock, staging swap, empty-dataset guard, CSV streaming, ingest_run provenance) will be carried over. Reference implementation kept on branch pg-write.