Merge pull request #325 from bigbio/dev

release: drift probes, HPC container path, and README deduplication

Author: enriquea

README.md

@@ -23,32 +23,17 @@ eval "$(poetry env activate)"
 
 ### Optional extras
 
-The base install is intentionally lean — heavy plotting and ML dependencies are
-opt-in via Poetry extras. Install only what a given workflow needs:
-
-| Extra | Enables | Pulls in |
-| --- | --- | --- |
-| `viz` | Static and interactive plotting | matplotlib, seaborn, plotly |
-| `interactive` | Interactive plots / dashboards | plotly |
-| `duckdb` | DuckDB-backed queries | duckdb |
-| `hgc` | Joint genotyping plots (genotype adjustment needs no extra) | matplotlib, seaborn |
-| `psroc` | Pathogenicity Score ROC analysis | matplotlib, plotly, scikit-learn, scipy |
-| `ptm` | CPTAC proteomics builders | cptac, sorted-nearest |
-| `constraint` | Tissue-specificity / constraint metrics | tspex, matplotlib, seaborn, scipy |
-| `enrichex` | `hvantk enrichex overlap` / `burden` and their plots | scipy, matplotlib, seaborn |
-| `cohort` | `hvantk cohort burden` (Fisher gene burden) | scipy |
-| `ancestry` | Ancestry inference (PCA + Random Forest + plots) | scikit-learn, matplotlib, seaborn, scipy |
-| `ml` | scikit-learn-backed features only | scikit-learn, scipy |
-| `expression` | `hvantk expression summarize` / `markers`, and `ptm constraint --expression-metric mean` | scanpy, scipy |
+The base install is intentionally lean — plotting, machine-learning, and a few
+provider-specific dependencies are opt-in Poetry extras:
 
 ```bash
-# One or more extras at once
-poetry install --extras "ancestry psroc"
-
-# Or a single extra
-poetry install --extras ml
+poetry install --extras "ancestry psroc"   # one or more
+poetry install --all-extras                # everything
 ```
 
+For the full table — what each extra pulls in and which commands need it — see
+[Installation → Optional features](docs_site/getting-started/installation.md#optional-features-extras).
+
 Verify it works:
 
 ```bash
@@ -122,105 +107,24 @@ returns `(native_obj, Provenance)` zero-cost.
 
 ### Plugin contract — adding a new data source
 
-Each plugin under `hvantk/skills/<plugin>/` declares itself via
-[`plugin.yaml`](hvantk/skills/clinvar/plugin.yaml) and provides a builder
-that returns a typed artifact:
-
-```python
-# hvantk/skills/clinvar/builder.py
-def build_clinvar(parsed_input, ctx: BuildContext, **params) -> AnnotationTable:
-    ht = hl.import_vcf(str(parsed_input), force=True, ...).rows().key_by("locus", "alleles")
-    return AnnotationTable.from_hail(
-        ht, provenance=ctx.provenance(schema_id="clinvar-variants-v1")
-    )
-```
-
-The platform orchestrator [`run_builder_for_spec`](hvantk/core/plugin/run_builder.py)
-ties it all together at build time:
-
-```mermaid
-sequenceDiagram
-    participant CLI as hvantk reprocess
-    participant Reg as plugin registry
-    participant Probe as drift_probe()
-    participant Build as build_fn(parsed, ctx)
-    participant IO as core/io
-
-    CLI->>Reg: get_dataset("clinvar:variants")
-    Reg-->>CLI: DatasetSpec (lazy bind on first access)
-    CLI->>Probe: compute source fingerprint
-    Probe-->>CLI: probe dict
-    CLI->>CLI: BuildContext(plugin, version, fingerprint, …)
-    CLI->>Build: (parsed_input, ctx, **params)
-    Build-->>CLI: AnnotationTable(provenance=ctx.provenance(schema_id=…))
-    CLI->>CLI: validate artifact_type + schema_id
-    CLI->>IO: artifact.save(path)
-    IO-->>IO: write data + sidecar .provenance.json
-```
-
-Twenty-one plugins ship today: `clinvar`, `clingen`, `gencc`, `gwas-catalog`,
-`hgnc`, `gtex-eqtl`, `insider`, `msigdb`, `uniprot-ptm`, `peptideatlas`,
-`expression-atlas`, `cptac`, `ucsc-cellbrowser`, `gevir`, `gnomad-metrics`,
-`ensembl-gene`, `dbnsfp`, `cosmic-cgc`, `pqtl`, `alphagenome`, `onek-genomes`.
-
-### Project structure
-
-```
-hvantk/
-├── core/                       # platform substrate — stable contracts
-│   ├── models/                 # AnnotationTable, ExpressionMatrix, VariantMatrix, GeneSet,
-│   │                           #   Provenance, BuildContext, Expr DSL,
-│   │                           #   AlgorithmMeta (@algorithm decorator)
-│   ├── io/                     # save / load / save_native / load_native,
-│   │                           #   sidecar provenance manifests, legacy shim
-│   ├── plugin/                 # plugin registry, run_builder_for_spec,
-│   │                           #   two-pass discovery (DatasetManifest → DatasetSpec)
-│   ├── tool/                   # tool manifest discovery (descriptive)
-│   ├── streamers/              # Streamer ABCs — query/iterate built tables
-│   │                           #   (concrete subclasses live in skills/<plugin>/)
-│   ├── ontology/               # OBO / MONDO parsers
-│   └── utils/                  # generic helpers (hail context, hail_helpers,
-│                               #   file utils, gene sets)
-│
-├── algorithms/                 # analytics — consume artifacts, return artifacts
-│   ├── ancestry/               # PCA + Random Forest ancestry inference
-│   ├── enrichex/               # gene set enrichment + burden testing
-│   ├── expression/             # tissue specificity (tau, gini, etc.)
-│   ├── hgc/                    # joint genotyping (gvcf combine, VDS, QC)
-│   ├── ptm/                    # PTM coordinate mapping + atlas
-│   ├── psroc/                  # pathogenicity score ROC analysis
-│   ├── qtlcascade/             # eQTL → pQTL cascade + colocalization
-│   ├── annotation/             # spine / prepare / compose annotation pipeline
-│   ├── burden/, cohort/        # rare-variant burden + external cohort handling
-│   ├── rerank/                 # multi-omic gene re-ranking (feature axes + audit)
-│   ├── statistics/             # multiple-testing correction, shared stats
-│   └── visualization/          # shared figure helpers (empty_figure, save_figure)
-│
-├── skills/                     # data-source plugins (21 total)
-│   ├── <plugin>/
-│   │   ├── plugin.yaml         # declarative manifest (drives discovery + CLI)
-│   │   ├── builder.py          # Phase B: (parsed, ctx) → AnnotationTable / …
-│   │   ├── drift_probe.py      # upstream fingerprint
-│   │   ├── cli.py              # downloader (auto-wired via manifest cli: block)
-│   │   └── tests/              # per-plugin conformance tests + fixtures
-│   └── _conventions/SKILL.md   # contract documentation
-│
-├── tools/                      # CLI wiring + workflow orchestration
-│   ├── plugins/                # download, drift, reprocess, plugins/tools list
-│   ├── hgc/                    # joint-genotyping CLI (lazy-loaded)
-│   ├── infra/                  # catalog, utils (check-install, bgzf)
-│   ├── annotation/             # annotate spine / prepare / compose
-│   ├── cohort/                 # cohort validate / burden / attach
-│   ├── rerank/                 # rerank CLI
-│   ├── genesets/               # gene set extraction / preparation
-│   ├── training_sets/          # TrainingSetBuilder — library only, no CLI command
-│   └── ancestry/, enrichex/, expression/, ptm/, qtl/   # one package per domain
-│
-├── resources/                  # platform metadata (unified catalog registry)
-└── tests/                      # cross-cutting tests (dependency directions,
-                                #   plugin conformance, io round-trips,
-                                #   Expr algebra parity, etc.)
-```
+Each data source ships as a self-contained plugin under `hvantk/skills/<plugin>/`,
+declared by a [`plugin.yaml`](hvantk/skills/clinvar/plugin.yaml) manifest naming its
+builder and drift probe, plus an optional downloader for sources that permit an
+automated fetch. Sources behind a license gate, or too large to mirror, ship a
+documented acquisition procedure instead. The platform orchestrator
+[`run_builder_for_spec`](hvantk/core/plugin/run_builder.py) resolves the manifest,
+computes the source fingerprint, calls the builder, validates the returned artifact
+against the manifest's `artifact_type` and `schema_id`, and saves it alongside a
+sidecar `.provenance.json`. The loader discovers manifests on its own — there is no
+registry to edit.
+
+The full contract and the annotated directory tree live in the architecture guide:
+
+- [Plugin contract](docs_site/architecture.md#3-plugin-contract--adding-a-data-source)
+  — build sequence diagram, annotated `plugin.yaml`, two-pass loader, streamer
+  placement rule
+- [Project structure](docs_site/architecture.md#project-structure) — what lives in
+  each package, layer by layer
 
 ### How to extend
 

containers/hvantk.def

@@ -0,0 +1,58 @@
+Bootstrap: docker
+From: python:3.10-slim-bullseye
+
+# hvantk container image, built from the committed poetry.lock
+# (hail 0.2.137 + pyspark 3.5.8). See docs_site/guide/hpc-migration.md section 3.
+#
+# BASE IMAGE: bullseye, not bookworm. Debian 12 (bookworm) has no openjdk-11 package
+# at all -- apt reports it is replaced by openjdk-17-jre-headless, and Hail 0.2.x /
+# Spark 3.5 support Java 8 or 11 ONLY. Debian 11 (bullseye) still ships
+# openjdk-11-jdk-headless. Do not "modernise" this base without checking `java -version`
+# inside the built image.
+#
+# EXTRAS: checked against [project.optional-dependencies] in pyproject.toml. This set
+# covers every unique package across all extras. `expression` is required or
+# `hvantk expression ...` has no scanpy. `ml` and `interactive` are deliberately
+# omitted as redundant: scikit-learn arrives via ancestry/psroc, plotly via viz.
+
+%files
+    pyproject.toml /opt/hvantk/pyproject.toml
+    poetry.lock    /opt/hvantk/poetry.lock
+    hvantk         /opt/hvantk/hvantk
+    README.md      /opt/hvantk/README.md
+
+%post
+    set -e
+    # --- Java 11 (Hail 0.2.137 / Spark 3.5 support Java 8 or 11 ONLY) + native libs ---
+    apt-get update && apt-get install -y --no-install-recommends \
+        openjdk-11-jdk-headless \
+        build-essential g++ \
+        zlib1g-dev libbz2-dev liblzma-dev libcurl4-openssl-dev libdeflate-dev \
+        libopenblas-dev liblapack-dev liblz4-dev libhdf5-dev git
+    # --- hvantk from the committed lock (reproducible; not unpinned pip install) ---
+    pip install --no-cache-dir "poetry>=2.0"
+    cd /opt/hvantk
+    poetry config virtualenvs.create false
+    poetry install --no-interaction --no-root \
+        --extras "hgc ptm ancestry psroc constraint enrichex cohort viz duckdb expression"
+    poetry install --no-interaction --only-root
+    apt-get purge -y build-essential g++ git && apt-get autoremove -y
+    apt-get clean && rm -rf /var/lib/apt/lists/*
+    # fail the build here rather than shipping a broken image
+    java -version 2>&1 | head -1
+    python -c "import hail; print('hail', hail.__version__)"
+
+%environment
+    export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
+    export PATH=$JAVA_HOME/bin:$PATH
+    export LC_ALL=C.UTF-8 LANG=C.UTF-8
+    export NO_PROXY=localhost,127.0.0.1,0.0.0.0,::1
+    export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1
+
+%runscript
+    exec hvantk "$@"
+
+%labels
+    org hvantk
+    hail 0.2.137
+    pyspark 3.5.8

containers/hvantk_run.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Run hvantk from the Apptainer/Singularity image with Spark scratch set up correctly.
+#
+# Without SPARK_LOCAL_DIRS on a writable NODE-LOCAL dir, Hail init inside the container
+# fails with "DiskBlockManager: Failed to create any local dir" followed by a misleading
+# "[Errno 111] Connection refused" from py4j. Always go through this wrapper.
+#
+#   bash hvantk_run.sh utils check-install
+#   bash hvantk_run.sh reprocess clinvar:variants --raw-dir ... --output ...
+#   HVANTK_SIF=/path/to/hvantk.sif bash hvantk_run.sh --help
+set -euo pipefail
+
+HVANTK_SIF="${HVANTK_SIF:-${WORK:?WORK unset}/containers/hvantk.sif}"
+[[ -f "$HVANTK_SIF" ]] || { echo "ERROR: image not found: $HVANTK_SIF" >&2; exit 1; }
+
+# node-local scratch, never home/Lustre/GPFS (guide section 4.1)
+SPARK_SCRATCH="${SLURM_TMPDIR:-/tmp/spark-$USER-${SLURM_JOB_ID:-$$}}"
+mkdir -p "$SPARK_SCRATCH"
+cleanup(){ rm -rf "$SPARK_SCRATCH"; }
+trap cleanup EXIT
+
+# Bind any extra data roots the command needs, e.g. HVANTK_BIND="$WORK:$WORK"
+BINDS=(-B "$SPARK_SCRATCH:$SPARK_SCRATCH")
+[[ -n "${HVANTK_BIND:-}" ]] && BINDS+=(-B "$HVANTK_BIND")
+
+exec singularity exec "${BINDS[@]}" \
+  --env SPARK_LOCAL_DIRS="$SPARK_SCRATCH" \
+  --env TMPDIR="$SPARK_SCRATCH" \
+  --env HAIL_TMPDIR="$SPARK_SCRATCH" \
+  "$HVANTK_SIF" hvantk "$@"

docs_site/architecture.md

@@ -294,6 +294,13 @@ def build_my_source_variants(parsed_input, ctx: BuildContext, **params) -> Annot
     )
 ```
 
+A shipped one to copy from: `hvantk/skills/clinvar/builder.py` defines
+`build_clinvar`, which `hvantk/skills/clinvar/plugin.yaml` binds to the
+`clinvar:variants` dataset under schema `clinvar-variants-v1`. The builder name
+is whatever `plugin.yaml`'s `builder.function` declares — plugins in the tree use
+both the short form (`build_clinvar`) and the per-dataset form
+(`build_clingen_gene_disease`).
+
 The plugin loader (`hvantk/core/plugin/loader.py`) discovers manifests via a
 **two-pass mechanism**:
 

docs_site/guide/data-sources.md

@@ -179,12 +179,22 @@ hvantk reprocess gnomad-metrics:metrics --skip-download \
   --plugin-arg key=transcript
 ```
 
-### INSIDER interactome (~100 MB)
+### INSIDER interactome (~1.2 GB genomic BED; ~49 MB pair table)
 
-Protein-protein interaction sites from the INSIDER database.
+Protein-protein interaction interface residues from the INSIDER database.
 URL: http://interactomeinsider.yulab.org/downloads.html
 
-**Download**: Visit http://interactomeinsider.yulab.org/downloads.html and download the interaction site BED file.
+INSIDER ships **two** products, and hvantk builds a dataset from each:
+
+| dataset | file | size | direct URL |
+| --- | --- | --- | --- |
+| `insider:variants` | `Whole_Human_Interactome_Interface_hg38.bed` | ~1.17 GB | `http://interactomeinsider.yulab.org/bed/all.bed` |
+| `insider:interfaces` | `H_sapiens_interfacesALL.txt` | ~49 MB | `http://interactomeinsider.yulab.org/downloads/interfacesALL/H_sapiens_interfacesALL.txt` |
+
+**Download**: the downloads page carries no links in its markup, so use the direct
+URLs above (they are also recorded in the plugin catalog — `hvantk catalog show
+INSIDER_v1.0`). Both are served over plain HTTP; the site has no HTTPS listener.
+The BED is >1 GB, so acquisition is manual per the downloader framework in CLAUDE.md.
 
 **Build**:
 
@@ -239,11 +249,25 @@ pathogenicity score. Abramovs, Brass & Tassabehji, 2020, Nature Genetics
 52(1):35-39 (PMID 31873297, DOI 10.1038/s41588-019-0560-2).
 URL: https://www.nature.com/articles/s41588-019-0560-2
 
-**Download**: Small supplementary table from the Nature Genetics publication
-(https://www.nature.com/articles/s41588-019-0560-2) or the authors' repository
-(https://github.com/gevirank/gevir). At ~1-2 MB with a stable, public URL, GeVIR
-qualifies for a real downloader under the framework in CLAUDE.md — a recommended
-follow-up (not yet implemented).
+**Download**: the metric table is **Supplementary Table 2** of the Nature Genetics
+paper, served as the article's MOESM3 object:
+
+```
+https://static-content.springer.com/esm/art%3A10.1038%2Fs41588-019-0560-2/MediaObjects/41588_2019_560_MOESM3_ESM.xlsx
+```
+
+That is a ~10.3 MB `.xlsx` workbook (only MOESM3 of the six supplementary slots is
+public; the rest return 403). The builder reads a bgzipped TSV, so extract sheet
+`table_2` and BGZF-compress it before building.
+
+> **Note:** the authors' repository at https://github.com/gevirank/gevir ships the
+> **analysis code only** — its `tables/` directory holds a placeholder file — so it
+> is not a source for the metric table. Earlier revisions of this guide pointed
+> there.
+
+A real downloader would have to do the extract-and-convert step, not just fetch the
+URL, so it is more than the usual thin wrapper — a recommended follow-up (not yet
+implemented).
 
 **Build**: