Pigeon iterative MAG binning + isolate assembly by microbemarsh · Pull Request #127 · treangenlab/somatem

microbemarsh · 2026-05-17T01:53:57Z

Bioconda cleanup, launcher fixes, and Pigeon iterative MAG binning

Summary

This branch cleans up the local/Bioconda Somatem experience and hardens the MAG assembly workflow. It fixes packaged launcher shortcuts, adds local CPU/memory detection, updates problematic module environments, and introduces Pigeon-driven iterative MAG binning as the default binning route.

Related to #109, #90 (fully fixed), and partially fixed #41.

This was getting pretty long and I decided it was worthy of a merge before I kept adding more.

Key Changes

Fixed bin/somatem shortcut commands so packaged utility workflows resolve correctly.
Added local resource detection and caps for high-resource processes.
Replaced the default MAG binning path with PIGEON_ITERATIVE_BINNING, while preserving the original SemiBin2-only path with mag_iterative_binning_enabled: false.
Added bin/pigeon_iterate.py to run SemiBin2, MetaBAT2, and VAMB across bounded iterations, select candidates with Pigeon metrics, and run DAS Tool consensus.
Improved Pigeon scoring with an adaptive graph-aware loss objective.
Added iterative_report.html plus TSV/JSON selection outputs so users can inspect Pigeon’s candidate trajectory and selected bins.
Published iterative binning, taxonomy, quality, assembly, and annotation outputs into user-facing result folders.
Fixed Hostile staged index handling, minimap2 local resource failures, DAS Tool compressed assembly handling, VAMB residual mismatch handling, and the Bakta 1.11.x annotation crash.
Removed the experimental CheckM2-in-Pigeon scoring path; downstream CheckM2 quality assessment remains unchanged.

Validation

Tested this on villapol lab PC with the example mag dataset.

The MAG workflow was run on the zymo example dataset to debug Hostile, minimap2, VAMB, DAS Tool, Bakta, and Pigeon output-publishing issues.

Reviewer Notes

mag_iterative_binning_enabled: true now sends MAG bins through Pigeon iterative binning by default.
mag_iterative_binning_enabled: false keeps the standard SemiBin2-only route available.
The Pigeon HTML report uses Plotly when available and falls back to table-only HTML if needed.

ppreshant · 2026-05-17T03:41:08Z

Woohoo! This looks great. I'll check and merge by Mon afternoon unless you want it earlier.

…

On Sat, May 16, 2026, 8:54 PM Austin Marshall ***@***.***> wrote: @microbemarsh <https://github.com/microbemarsh> requested your review on: treangenlab/somatem#127 <#127> Bioconda cleanup i109. — Reply to this email directly, view it on GitHub <#127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRLCHYOIESKP4KUVTCLPQ343ELUVAVCNFSM6AAAAACZBAHAESVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRVGYYTQNBSGYZTSNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because your review was requested.Message ID: ***@***.***>

microbemarsh · 2026-05-22T03:51:33Z

Summary of recent commit:

Big points - added isolate_analysis.nf and created summary_report module for pipeline summaries

Nitty gritty AI gen from the git commit below:

Added shared SOMaTeM HTML summary reporting infrastructure via bin/somatem_report.py and a reusable SOMATEM_SUMMARY_REPORT module.
Wired summary reports into pre-processing, taxonomic profiling, MAG assembly, genome dynamics, and isolate analysis workflows.
Added a new long-read-first isolate-analysis workflow that consumes preprocessed reads.
Added isolate read classification with Kraken2, assembly with Autocycler/Flye, optional hybrid polishing with Polypolish and pypolca, Bakta annotation, and optional BTyper3 typing.
Added tool-first local modules and Conda environments for Autocycler, Flye-for-Autocycler, BWA, Polypolish, pypolca, BTyper3, and FASTA finalization.
Reused and updated existing Bakta and Kraken2 modules for isolate workflow outputs and version reporting.
Added isolate_analysis CLI support through bin/somatem.
Updated database handling to support isolate-analysis Bakta DB setup.
Updated EMU database Add latest emu and lemur databases (2025) from Mike #56 download/staging logic and default EMU DB config.
Improved example dataset download behavior with configurable output location and completion tracking.
Added Conda environment stability tweaks, including longer create timeouts and nodefaults for Hostile environments.

ppreshant

Great updates. Other than reverting the path changes in the example csv and yaml files everything looks good to be merged.

If you're unable to get to the path changes, I can do them early next week.

ppreshant · 2026-05-23T05:34:38Z

@@ -1,2 +1,2 @@
 sample,fastq_1
-zymo,assets/data/assembly/mock20_hiq100k.fastq.gz
+zymo,/home/agm/Documents/somatem/assets/data/assembly/mock20_hiq100k.fastq.gz


These paths are hardcoded to specific directoreis and will fail for users. The original path assets/data/assembly/mock20_hiq100k.fastq.gz works when the users download example files using the script: subworkflows/local/download_example_datasets.nf saving files here:

params.save_dir = params.save_dir ?: "${launchDir}/assets" // default save directory: // currently users can't change this since the config yml files are hardcoded to use this path

I will revert this and other similar changes manually for now

ppreshant · 2026-05-23T05:35:45Z

 # 1. laptop = runs on a laptop. Only does taxonomic profiling (default)
 # 2. cluster = runs on a cluster or workstation. Can perform all analysis_types (including assembly)
-running_mode: "cluster"
+running_mode: "laptop"


I think the running_mode is not used currently so this is redundant and can be removed from these params to minimize distraction

ppreshant · 2026-05-23T05:41:43Z

 ### Set the cache dir
-Nextflow's cache dir is set to `~/micromamba/other-envs` in the `nextflow.config` file's `conda.cacheDir` parameter. So create a directory at this location the first time.
-_The default cache dir for nextflow created environments is `Somatem/work/..`_; Since I like to delete `work/` frequently to save space, I set this to a different location outside the repo.
+Nextflow's conda cache and downloaded databases default to the Somatem data directory. In a Bioconda or micromamba install this resolves to `$CONDA_PREFIX/var/somatem`, so generated environments and databases stay with the active Somatem environment.


Actually now that I rethink, this will cause issues by storing other environments within the somatem environment. Nextflow activates and switches between them independently so it is best to leave it at the default location of conda.cacheDir at $workDir/conda path and mention this for power users' to change.

I think the nextflow config features an automatic detection of somatem home and setting the conda or mamba path based directory for the environments so this documentation might need to be updated?

ppreshant · 2026-05-23T05:44:04Z

+Nextflow's conda cache and downloaded databases default to the Somatem data directory. In a Bioconda or micromamba install this resolves to `$CONDA_PREFIX/var/somatem`, so generated environments and databases stay with the active Somatem environment.

 ```sh
-mkdir -p ~/micromamba/other-envs


Not sure if this cache directory is automatically made by nextflow?

If not, we should have a setup nextflow module that makes this dir rather than have users make it?

ppreshant · 2026-05-23T05:46:03Z


    // Run SingleM pipe on bins - FIXED: Add suffix to avoid filename collision
-    SINGLEM_PIPE_BINS(SEMIBIN_SINGLEEASYBIN.out.output_fasta, ch_singlem_db, 'genome')
+    SINGLEM_PIPE_BINS(ch_bins_for_downstream, ch_singlem_db, 'genome')


Minor note (for future): ch_bins_for_downstream rename to more descriptive? Since it is used in many places in the pipeline, it would make it clearer by being self-documenting!

ppreshant · 2026-05-23T05:48:51Z

    input                      = "${projectDir}/assets/samplesheet.csv" // bundled default file should resolve from the installed pipeline root
-    db_base_dir                = "/home/dbs" // change to "./assets/databases" if not using a shared directory
-    unified_db_base_dir        = "/home/Users/pacbio_bakeoff/data/ref_db/refseq03032025" // path to Eddy's unified databases (Ensemble analysis: species detection)
+    somatem_home               = System.getenv('SOMATEM_HOME') ?: ( System.getenv('CONDA_PREFIX') ? "${System.getenv('CONDA_PREFIX')}/var/somatem" : ( System.getenv('MAMBA_ROOT_PREFIX') ? "${System.getenv('MAMBA_ROOT_PREFIX')}/var/somatem" : "${System.getenv('HOME')}/.somatem" ) )


Interesting method!

ppreshant · 2026-05-23T11:56:36Z

@copilot please address my comments from the review in a few commits!

Agent-Logs-Url: https://github.com/treangenlab/somatem/sessions/c367543a-34f2-4367-8c5e-6a497b81b5fd Co-authored-by: ppreshant <14856479+ppreshant@users.noreply.github.com>

Copilot · 2026-05-23T11:59:04Z

@copilot please address my comments from the review in a few commits!

Addressed in commit 1c7ca47. I reverted hardcoded local paths in the MAG example sample/config files, removed the unused running_mode entry from assets/user_config/mag_params.yml, and updated docs/installation.md cache guidance to default Nextflow behavior plus override env vars. No UI changes were made in this update.

microbemarsh added 2 commits May 16, 2026 09:52

add pigeon updates and revise default envs / dbs path

ee9c7fa

add pigeon loss fxn, fix publish, fix bakta

c418f70

microbemarsh added this to the v1 release: somatem pipeline on bioconda and for preprint milestone May 17, 2026

microbemarsh requested a review from ppreshant May 17, 2026 01:53

microbemarsh added the enhancement New feature or request label May 17, 2026

microbemarsh self-assigned this May 17, 2026

microbemarsh added the bug Something isn't working label May 17, 2026

microbemarsh marked this pull request as draft May 21, 2026 15:44

big: update emu db, add summary report, add isolate subwrkflw

188e84a

microbemarsh marked this pull request as ready for review May 22, 2026 13:39

ppreshant reviewed May 23, 2026

View reviewed changes

ppreshant changed the title ~~Bioconda cleanup i109~~ Pigeon iterative MAG binning + isolate assembly May 23, 2026

Copilot started work on behalf of ppreshant May 23, 2026 11:56 View session

Revert hardcoded example paths and clean MAG params/docs

1c7ca47

Agent-Logs-Url: https://github.com/treangenlab/somatem/sessions/c367543a-34f2-4367-8c5e-6a497b81b5fd Co-authored-by: ppreshant <14856479+ppreshant@users.noreply.github.com>

Copilot finished work on behalf of ppreshant May 23, 2026 11:59

Copilot AI requested a review from ppreshant May 23, 2026 11:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pigeon iterative MAG binning + isolate assembly #127

Pigeon iterative MAG binning + isolate assembly #127
microbemarsh wants to merge 4 commits into
mainfrom
bioconda_cleanup_i109

microbemarsh commented May 17, 2026 •

edited

Loading

Uh oh!

ppreshant commented May 17, 2026 via email

Uh oh!

microbemarsh commented May 22, 2026 •

edited

Loading

Uh oh!

ppreshant left a comment

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant May 23, 2026

Uh oh!

ppreshant commented May 23, 2026

Uh oh!

Copilot AI commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

microbemarsh commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bioconda cleanup, launcher fixes, and Pigeon iterative MAG binning

Summary

Key Changes

Validation

Reviewer Notes

Uh oh!

ppreshant commented May 17, 2026 via email

Uh oh!

microbemarsh commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ppreshant left a comment

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant May 23, 2026

Choose a reason for hiding this comment

Uh oh!

ppreshant commented May 23, 2026

Uh oh!

Copilot AI commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

microbemarsh commented May 17, 2026 •

edited

Loading

microbemarsh commented May 22, 2026 •

edited

Loading