Add explicit sequence-selection manifests - #222
Open
Megachile wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Highly fragmented assemblies explode in compute time for minimal gene gain. This allows users to preselect scaffolds to predict, eg filtering out massive numbers of small fragments or retaining only fragments meeting certain criteria.
Allow explicit sequence selection with a workload preview
Summary
This change lets users provide an explicit per-sequence selection before
FASTA-to-HDF5 conversion:
--sequence-selection selection.tsv
--selection-report selection_report.tsv
--selection-only
The selection file states which FASTA records to include. Helixer does not
decide the selection criterion: users may base it on sequence length,
placement, coverage, contamination review, manual curation, or any other
external evidence.
Without a selection file, Helixer behaves exactly as it does now.
Why
Helixer creates fixed-length inputs independently for every FASTA record and
both strands. In a highly fragmented assembly, a short contig can therefore
require far more model input than its sequence length would suggest. With
hundreds of thousands of records, padding can dominate runtime.
Users can create a filtered FASTA themselves, but a native selection interface
has three advantages:
it avoids requiring users to create and manage another filtered FASTA;
it records exactly which sequences were omitted and why;
it can report the resulting padded workload before compute begins.
Selection format
A minimal tab-separated format is sufficient:
seqid include reason
scaffold_1 true placed
scaffold_2 true length_at_least_5000
scaffold_3 false below_user_threshold
reason is optional provenance and has no effect on Helixer behavior.
Validation is strict:
every input FASTA ID must have exactly one decision;
duplicate, missing, and unknown IDs are errors;
input sequence order is preserved;
an empty selected set fails safely.
This keeps the interface criterion-agnostic. Placement from a scaffolder is
only one possible source of a decision, not a dependency or special case.
Workload preview
Before export or inference, Helixer reports:
full and selected records and bases;
full and selected fixed-window rows across both strands;
full and selected padded positions;
counts by selection reason;
a checksum of the selection file.
--selection-only validates the selection and writes this report without
running inference. The report describes tensor workload, not guaranteed wall
time.
Important limitation
This option changes annotation scope. Excluded sequences receive no
predictions. It must therefore be default-off and accompanied by a clear
warning; it must not be presented as a quality-neutral whole-genome
optimization.
In our motivating 1.57-Gb, 413,000-record assembly, two example selections
reduced fixed-window rows by 89–94% and completed on one H100 in approximately
14 and 10 hours. They retained 81–85% of accepted models and 93–95% of protein
sequence from the complete annotation. These values illustrate the tradeoff;
they are not proposed thresholds or expected results for other assemblies.
Predictions were exactly identical on sequences shared by the two selections
when run in the same execution environment. The cost came from omitted
sequences, not altered predictions on retained sequences.
Provenance
The selection report and selection-file checksum should be retained with the
run outputs so downstream users can distinguish a selected-scope annotation
from a complete-input annotation.
Tests
No selection file reproduces current behavior.
Missing, duplicate, and unknown sequence IDs.
All-included, partially included, and empty selections.
Exact selected-record order and sequence checksums.
Exact row and padding accounting.
--selection-only launches no inference.
Selection provenance is retained with outputs.
Nested selections produce identical predictions on shared sequences under
the same execution environment.
Scope
This change does not:
define or recommend a biological filtering criterion;
depend on RagTag or another scaffolder;
introduce a default minimum sequence length;
classify contamination;
claim that excluded sequences lack useful genes.