Question: support for workflows outside of spack?

[vsoch]

Hiya! Is there support (or planned support) for being able to define an execute workflows with other kinds of package managers? E.g., one very easy family would be something like conda/mamba to install a bunch of python machine learning libraries, which aren't handled as well under spack. Thanks!

[douglasjacobsen]

Hey @vsoch!

The infrastructure should support workflows like this. We don't have plans to implement them, but the way we've written stuff so far it could easily extend this.

I think if we had a few examples, it might not take much to implement. But, in case this helps show where the hooks might exist...

lib/ramble/ramble/application_types houses specific types of applications. Right now, they are:

• executable: Intended to be an executable application that can be assumed to be on the system through some other mechanism
• spack: Intended to be an executable that needs to be installed through spack

At a minimum, an executable application could be written (with some semi-manual installation steps in the documentation) to run some ML workloads. But, a separate application class (i.e. conda.py) could be written to automate some of the conda installation in venvs or something.

The other bit that's specific to spack is lib/ramble/ramble/spack_runner.py which handles calling out to execute spack commands and generate spack environments. Something similar would need to be implemented for conda too.

[vsoch]

Cool - thank you! Pinging @tpatki - I think we could probably do well to add some python-based ML type workflows or simulations. If you like this idea, I can go ahead and try adding.

Also pinging @wolfv - would this be a good place to sneak in pixi? Do you have Python bindings? It would be cool to test it out in a workflow context.

[wolfv]

That would be cool indeed! Not sure if pixi is at the right abstraction level (since it is kinda opinionated), but building on top of the rattler-library (and the upcoming Python bindings) would be very interesting!

[vsoch]

What about trying conda or mamba in the meantime? I think most workflows I have in mind use something like pytorch, so maybe (despite the TBA rattler / pixi being exciting) we could do that in the meantime. Snakemake, for example, just calls conda and I believe it's the most popular execution mode!

I'll also wait to hear from @tpatki on her thoughts!

[wolfv]

Yes, doing that with micromamba should be fairly straightforward, indeed! micromamba makes it much easier to set up one (or more) environments in an arbitrary location.

[douglasjacobsen]

Honestly, another pretty easy way to do it would be to use the spack package definition to install micromamba or miniconda, and then make the installation of your packages part of the executable steps.

I'd be happy to give some input if you have a concrete example for me to look at. :)

[vsoch]

@douglasjacobsen haha I think that would make me run away :) I have a few things on my queue that I want to work on first, but I'll come back to this to work on soon!