Skip to content

release: drift probes, HPC container path, and README deduplication #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
enriquea merged 4 commits into from
Sep 1, 2026
Merged

release: drift probes, HPC container path, and README deduplication #325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
420be1e
fix: real drift probes for gevir and insider, with review fixes (#319)
enriquea Sep 1, 2026
9b19484
feat: real drift probes for the five remaining documentation-only plu…
enriquea Sep 1, 2026
0bb8d94
docs(hpc): fix the container build path so it actually works on a clu…
enriquea Sep 1, 2026
c2b5d42
docs: fold the README's duplicated sections into the docs site (#324)
enriquea Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 25 additions & 121 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,32 +23,17 @@ eval "$(poetry env activate)"

### Optional extras

The base install is intentionally lean — heavy plotting and ML dependencies are
opt-in via Poetry extras. Install only what a given workflow needs:

| Extra | Enables | Pulls in |
| --- | --- | --- |
| `viz` | Static and interactive plotting | matplotlib, seaborn, plotly |
| `interactive` | Interactive plots / dashboards | plotly |
| `duckdb` | DuckDB-backed queries | duckdb |
| `hgc` | Joint genotyping plots (genotype adjustment needs no extra) | matplotlib, seaborn |
| `psroc` | Pathogenicity Score ROC analysis | matplotlib, plotly, scikit-learn, scipy |
| `ptm` | CPTAC proteomics builders | cptac, sorted-nearest |
| `constraint` | Tissue-specificity / constraint metrics | tspex, matplotlib, seaborn, scipy |
| `enrichex` | `hvantk enrichex overlap` / `burden` and their plots | scipy, matplotlib, seaborn |
| `cohort` | `hvantk cohort burden` (Fisher gene burden) | scipy |
| `ancestry` | Ancestry inference (PCA + Random Forest + plots) | scikit-learn, matplotlib, seaborn, scipy |
| `ml` | scikit-learn-backed features only | scikit-learn, scipy |
| `expression` | `hvantk expression summarize` / `markers`, and `ptm constraint --expression-metric mean` | scanpy, scipy |
The base install is intentionally lean — plotting, machine-learning, and a few
provider-specific dependencies are opt-in Poetry extras:

```bash
# One or more extras at once
poetry install --extras "ancestry psroc"

# Or a single extra
poetry install --extras ml
poetry install --extras "ancestry psroc" # one or more
poetry install --all-extras # everything
```

For the full table — what each extra pulls in and which commands need it — see
[Installation → Optional features](docs_site/getting-started/installation.md#optional-features-extras).

Verify it works:

```bash
Expand Down Expand Up @@ -122,105 +107,24 @@ returns `(native_obj, Provenance)` zero-cost.

### Plugin contract — adding a new data source

Each plugin under `hvantk/skills/<plugin>/` declares itself via
[`plugin.yaml`](hvantk/skills/clinvar/plugin.yaml) and provides a builder
that returns a typed artifact:

```python
# hvantk/skills/clinvar/builder.py
def build_clinvar(parsed_input, ctx: BuildContext, **params) -> AnnotationTable:
ht = hl.import_vcf(str(parsed_input), force=True, ...).rows().key_by("locus", "alleles")
return AnnotationTable.from_hail(
ht, provenance=ctx.provenance(schema_id="clinvar-variants-v1")
)
```

The platform orchestrator [`run_builder_for_spec`](hvantk/core/plugin/run_builder.py)
ties it all together at build time:

```mermaid
sequenceDiagram
participant CLI as hvantk reprocess
participant Reg as plugin registry
participant Probe as drift_probe()
participant Build as build_fn(parsed, ctx)
participant IO as core/io

CLI->>Reg: get_dataset("clinvar:variants")
Reg-->>CLI: DatasetSpec (lazy bind on first access)
CLI->>Probe: compute source fingerprint
Probe-->>CLI: probe dict
CLI->>CLI: BuildContext(plugin, version, fingerprint, …)
CLI->>Build: (parsed_input, ctx, **params)
Build-->>CLI: AnnotationTable(provenance=ctx.provenance(schema_id=…))
CLI->>CLI: validate artifact_type + schema_id
CLI->>IO: artifact.save(path)
IO-->>IO: write data + sidecar .provenance.json
```

Twenty-one plugins ship today: `clinvar`, `clingen`, `gencc`, `gwas-catalog`,
`hgnc`, `gtex-eqtl`, `insider`, `msigdb`, `uniprot-ptm`, `peptideatlas`,
`expression-atlas`, `cptac`, `ucsc-cellbrowser`, `gevir`, `gnomad-metrics`,
`ensembl-gene`, `dbnsfp`, `cosmic-cgc`, `pqtl`, `alphagenome`, `onek-genomes`.

### Project structure

```
hvantk/
├── core/ # platform substrate — stable contracts
│ ├── models/ # AnnotationTable, ExpressionMatrix, VariantMatrix, GeneSet,
│ │ # Provenance, BuildContext, Expr DSL,
│ │ # AlgorithmMeta (@algorithm decorator)
│ ├── io/ # save / load / save_native / load_native,
│ │ # sidecar provenance manifests, legacy shim
│ ├── plugin/ # plugin registry, run_builder_for_spec,
│ │ # two-pass discovery (DatasetManifest → DatasetSpec)
│ ├── tool/ # tool manifest discovery (descriptive)
│ ├── streamers/ # Streamer ABCs — query/iterate built tables
│ │ # (concrete subclasses live in skills/<plugin>/)
│ ├── ontology/ # OBO / MONDO parsers
│ └── utils/ # generic helpers (hail context, hail_helpers,
│ # file utils, gene sets)
│
├── algorithms/ # analytics — consume artifacts, return artifacts
│ ├── ancestry/ # PCA + Random Forest ancestry inference
│ ├── enrichex/ # gene set enrichment + burden testing
│ ├── expression/ # tissue specificity (tau, gini, etc.)
│ ├── hgc/ # joint genotyping (gvcf combine, VDS, QC)
│ ├── ptm/ # PTM coordinate mapping + atlas
│ ├── psroc/ # pathogenicity score ROC analysis
│ ├── qtlcascade/ # eQTL → pQTL cascade + colocalization
│ ├── annotation/ # spine / prepare / compose annotation pipeline
│ ├── burden/, cohort/ # rare-variant burden + external cohort handling
│ ├── rerank/ # multi-omic gene re-ranking (feature axes + audit)
│ ├── statistics/ # multiple-testing correction, shared stats
│ └── visualization/ # shared figure helpers (empty_figure, save_figure)
│
├── skills/ # data-source plugins (21 total)
│ ├── <plugin>/
│ │ ├── plugin.yaml # declarative manifest (drives discovery + CLI)
│ │ ├── builder.py # Phase B: (parsed, ctx) → AnnotationTable / …
│ │ ├── drift_probe.py # upstream fingerprint
│ │ ├── cli.py # downloader (auto-wired via manifest cli: block)
│ │ └── tests/ # per-plugin conformance tests + fixtures
│ └── _conventions/SKILL.md # contract documentation
│
├── tools/ # CLI wiring + workflow orchestration
│ ├── plugins/ # download, drift, reprocess, plugins/tools list
│ ├── hgc/ # joint-genotyping CLI (lazy-loaded)
│ ├── infra/ # catalog, utils (check-install, bgzf)
│ ├── annotation/ # annotate spine / prepare / compose
│ ├── cohort/ # cohort validate / burden / attach
│ ├── rerank/ # rerank CLI
│ ├── genesets/ # gene set extraction / preparation
│ ├── training_sets/ # TrainingSetBuilder — library only, no CLI command
│ └── ancestry/, enrichex/, expression/, ptm/, qtl/ # one package per domain
│
├── resources/ # platform metadata (unified catalog registry)
└── tests/ # cross-cutting tests (dependency directions,
# plugin conformance, io round-trips,
# Expr algebra parity, etc.)
```
Each data source ships as a self-contained plugin under `hvantk/skills/<plugin>/`,
declared by a [`plugin.yaml`](hvantk/skills/clinvar/plugin.yaml) manifest naming its
builder and drift probe, plus an optional downloader for sources that permit an
automated fetch. Sources behind a license gate, or too large to mirror, ship a
documented acquisition procedure instead. The platform orchestrator
[`run_builder_for_spec`](hvantk/core/plugin/run_builder.py) resolves the manifest,
computes the source fingerprint, calls the builder, validates the returned artifact
against the manifest's `artifact_type` and `schema_id`, and saves it alongside a
sidecar `.provenance.json`. The loader discovers manifests on its own — there is no
registry to edit.

The full contract and the annotated directory tree live in the architecture guide:

- [Plugin contract](docs_site/architecture.md#3-plugin-contract--adding-a-data-source)
— build sequence diagram, annotated `plugin.yaml`, two-pass loader, streamer
placement rule
- [Project structure](docs_site/architecture.md#project-structure) — what lives in
each package, layer by layer

### How to extend

Expand Down
58 changes: 58 additions & 0 deletions containers/hvantk.def
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
Bootstrap: docker
From: python:3.10-slim-bullseye

# hvantk container image, built from the committed poetry.lock
# (hail 0.2.137 + pyspark 3.5.8). See docs_site/guide/hpc-migration.md section 3.
#
# BASE IMAGE: bullseye, not bookworm. Debian 12 (bookworm) has no openjdk-11 package
# at all -- apt reports it is replaced by openjdk-17-jre-headless, and Hail 0.2.x /
# Spark 3.5 support Java 8 or 11 ONLY. Debian 11 (bullseye) still ships
# openjdk-11-jdk-headless. Do not "modernise" this base without checking `java -version`
# inside the built image.
#
# EXTRAS: checked against [project.optional-dependencies] in pyproject.toml. This set
# covers every unique package across all extras. `expression` is required or
# `hvantk expression ...` has no scanpy. `ml` and `interactive` are deliberately
# omitted as redundant: scikit-learn arrives via ancestry/psroc, plotly via viz.

%files
pyproject.toml /opt/hvantk/pyproject.toml
poetry.lock /opt/hvantk/poetry.lock
hvantk /opt/hvantk/hvantk
README.md /opt/hvantk/README.md

%post
set -e
# --- Java 11 (Hail 0.2.137 / Spark 3.5 support Java 8 or 11 ONLY) + native libs ---
apt-get update && apt-get install -y --no-install-recommends \
openjdk-11-jdk-headless \
build-essential g++ \
zlib1g-dev libbz2-dev liblzma-dev libcurl4-openssl-dev libdeflate-dev \
libopenblas-dev liblapack-dev liblz4-dev libhdf5-dev git
# --- hvantk from the committed lock (reproducible; not unpinned pip install) ---
pip install --no-cache-dir "poetry>=2.0"
cd /opt/hvantk
poetry config virtualenvs.create false
poetry install --no-interaction --no-root \
--extras "hgc ptm ancestry psroc constraint enrichex cohort viz duckdb expression"
poetry install --no-interaction --only-root
apt-get purge -y build-essential g++ git && apt-get autoremove -y
apt-get clean && rm -rf /var/lib/apt/lists/*
# fail the build here rather than shipping a broken image
java -version 2>&1 | head -1
python -c "import hail; print('hail', hail.__version__)"

%environment
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export LC_ALL=C.UTF-8 LANG=C.UTF-8
export NO_PROXY=localhost,127.0.0.1,0.0.0.0,::1
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1

%runscript
exec hvantk "$@"

%labels
org hvantk
hail 0.2.137
pyspark 3.5.8
30 changes: 30 additions & 0 deletions containers/hvantk_run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash
# Run hvantk from the Apptainer/Singularity image with Spark scratch set up correctly.
#
# Without SPARK_LOCAL_DIRS on a writable NODE-LOCAL dir, Hail init inside the container
# fails with "DiskBlockManager: Failed to create any local dir" followed by a misleading
# "[Errno 111] Connection refused" from py4j. Always go through this wrapper.
#
# bash hvantk_run.sh utils check-install
# bash hvantk_run.sh reprocess clinvar:variants --raw-dir ... --output ...
# HVANTK_SIF=/path/to/hvantk.sif bash hvantk_run.sh --help
set -euo pipefail

HVANTK_SIF="${HVANTK_SIF:-${WORK:?WORK unset}/containers/hvantk.sif}"
[[ -f "$HVANTK_SIF" ]] || { echo "ERROR: image not found: $HVANTK_SIF" >&2; exit 1; }

# node-local scratch, never home/Lustre/GPFS (guide section 4.1)
SPARK_SCRATCH="${SLURM_TMPDIR:-/tmp/spark-$USER-${SLURM_JOB_ID:-$$}}"
mkdir -p "$SPARK_SCRATCH"
cleanup(){ rm -rf "$SPARK_SCRATCH"; }
trap cleanup EXIT
Comment on lines +17 to +20

@qodo-code-review qodo-code-review Bot Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Scheduler scratch can be deleted 🐞 Bug ☼ Reliability

When SLURM_TMPDIR is set, the wrapper assigns the scheduler-owned directory itself to
SPARK_SCRATCH and recursively deletes it from the EXIT trap. Any failure after the trap is
installed, including a missing singularity executable, can erase unrelated files stored in the
job's shared scratch directory.
Agent Prompt 
## Issue description
The wrapper may recursively delete the complete scheduler-provided `SLURM_TMPDIR`, including files it did not create.

## Issue Context
`SLURM_TMPDIR` should be treated as a parent directory. Create a uniquely named child owned by this invocation, bind that child, and remove only that child. Account for the fact that a successful `exec` replaces the shell and does not execute its EXIT trap.

## Fix Focus Areas
- containers/hvantk_run.sh[16-30]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


# Bind any extra data roots the command needs, e.g. HVANTK_BIND="$WORK:$WORK"
BINDS=(-B "$SPARK_SCRATCH:$SPARK_SCRATCH")
[[ -n "${HVANTK_BIND:-}" ]] && BINDS+=(-B "$HVANTK_BIND")

exec singularity exec "${BINDS[@]}" \

@qodo-code-review qodo-code-review Bot Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Apptainer-only clusters cannot run 🐞 Bug ≡ Correctness

The wrapper advertises Apptainer/Singularity support but unconditionally invokes singularity exec.
On clusters exposing only the documented apptainer executable, every wrapper command fails before
starting hvantk.
Agent Prompt 
## Issue description
The wrapper hardcodes `singularity`, preventing execution on Apptainer-only clusters even though both runtimes are documented as supported.

## Issue Context
Resolve `apptainer` or `singularity` before creating scratch state, allow an explicit override if useful, and emit a clear error if neither exists.

## Fix Focus Areas
- containers/hvantk_run.sh[11-30]
- docs_site/guide/hpc-migration.md[93-104]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

--env SPARK_LOCAL_DIRS="$SPARK_SCRATCH" \
--env TMPDIR="$SPARK_SCRATCH" \
--env HAIL_TMPDIR="$SPARK_SCRATCH" \
"$HVANTK_SIF" hvantk "$@"
7 changes: 7 additions & 0 deletions docs_site/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,13 @@ def build_my_source_variants(parsed_input, ctx: BuildContext, **params) -> Annot
)
```

A shipped one to copy from: `hvantk/skills/clinvar/builder.py` defines
`build_clinvar`, which `hvantk/skills/clinvar/plugin.yaml` binds to the
`clinvar:variants` dataset under schema `clinvar-variants-v1`. The builder name
is whatever `plugin.yaml`'s `builder.function` declares — plugins in the tree use
both the short form (`build_clinvar`) and the per-dataset form
(`build_clingen_gene_disease`).

The plugin loader (`hvantk/core/plugin/loader.py`) discovers manifests via a
**two-pass mechanism**:

Expand Down
40 changes: 32 additions & 8 deletions docs_site/guide/data-sources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,22 @@ hvantk reprocess gnomad-metrics:metrics --skip-download \
--plugin-arg key=transcript
```

### INSIDER interactome (~100 MB)
### INSIDER interactome (~1.2 GB genomic BED; ~49 MB pair table)

Protein-protein interaction sites from the INSIDER database.
Protein-protein interaction interface residues from the INSIDER database.
URL: http://interactomeinsider.yulab.org/downloads.html

**Download**: Visit http://interactomeinsider.yulab.org/downloads.html and download the interaction site BED file.
INSIDER ships **two** products, and hvantk builds a dataset from each:

| dataset | file | size | direct URL |
| --- | --- | --- | --- |
| `insider:variants` | `Whole_Human_Interactome_Interface_hg38.bed` | ~1.17 GB | `http://interactomeinsider.yulab.org/bed/all.bed` |
| `insider:interfaces` | `H_sapiens_interfacesALL.txt` | ~49 MB | `http://interactomeinsider.yulab.org/downloads/interfacesALL/H_sapiens_interfacesALL.txt` |

**Download**: the downloads page carries no links in its markup, so use the direct
URLs above (they are also recorded in the plugin catalog — `hvantk catalog show
INSIDER_v1.0`). Both are served over plain HTTP; the site has no HTTPS listener.
The BED is >1 GB, so acquisition is manual per the downloader framework in CLAUDE.md.

**Build**:

Expand Down Expand Up @@ -239,11 +249,25 @@ pathogenicity score. Abramovs, Brass & Tassabehji, 2020, Nature Genetics
52(1):35-39 (PMID 31873297, DOI 10.1038/s41588-019-0560-2).
URL: https://www.nature.com/articles/s41588-019-0560-2

**Download**: Small supplementary table from the Nature Genetics publication
(https://www.nature.com/articles/s41588-019-0560-2) or the authors' repository
(https://github.com/gevirank/gevir). At ~1-2 MB with a stable, public URL, GeVIR
qualifies for a real downloader under the framework in CLAUDE.md — a recommended
follow-up (not yet implemented).
**Download**: the metric table is **Supplementary Table 2** of the Nature Genetics
paper, served as the article's MOESM3 object:

```
https://static-content.springer.com/esm/art%3A10.1038%2Fs41588-019-0560-2/MediaObjects/41588_2019_560_MOESM3_ESM.xlsx
```

That is a ~10.3 MB `.xlsx` workbook (only MOESM3 of the six supplementary slots is
public; the rest return 403). The builder reads a bgzipped TSV, so extract sheet
`table_2` and BGZF-compress it before building.

> **Note:** the authors' repository at https://github.com/gevirank/gevir ships the
> **analysis code only** — its `tables/` directory holds a placeholder file — so it
> is not a source for the metric table. Earlier revisions of this guide pointed
> there.

A real downloader would have to do the extract-and-convert step, not just fetch the
URL, so it is more than the usual thin wrapper — a recommended follow-up (not yet
implemented).

**Build**:

Expand Down
Loading
Loading