Generate working containers from Neurodocker recipes

[Vbitz]

Neurodocker (https://github.com/ReproNim/neurodocker) is a Python-based tool for easily generating Singularity and Docker container definitions from shell scripts. Pkg2 has had support for parsing these scripts for a while and supports extracting dependency information. With the recent addition of code to export docker containers I'd like to test generating a working container from these "definitions".

I'll start with the recipes defined in https://github.com/NeuroDesk/neurocontainers

[Vbitz]

This depends on #8. The previous version worked but it relied on some shell syntax being stubbed which is no longer the case.

This commit updates the script to the latest version e7f7daf

[Vbitz]

I've got the relevant data extracted from the builder but there are two problems.

• The directives are in the wrong order. I currently defer all NeuroDocker packages to after the rest of the directives are evaluated.
• I can't resolve a installation plan for the packages. The error is failed to run script: failed to add package for fedora@35/cmake@3.24.2:x86_64: failed to add package for fedora@35/cmake-data@3.24.2:any: failed to add package for fedora@35/cmake-rpm-macros@3.24.2:any: failed to add package for fedora@35/python3@3.10.8:i686: failed to add package for fedora@35/python3-libs@3.10.8:i686: could not find installation candidate among options: [fedora@35/libtinfo.so.6@:].
  • I think the error is I'm not resolving the name for libtinfo correctly.

[Vbitz]

The other issue is I'm resolving i686 packages when the install was requested for x86_64. I need to propagate the architecture into names unless a override is set for some reason.

[Vbitz]

d489794 Fixes that issue among others. I've now got a installation plan for afni.

[Vbitz]

ce11984 fetches the contents of the RPM archives. Looking at the basic info it seems like it will be similar code to the Alpine docker image generator (It uses pre and post install scripts, I haven't seen any mentions of triggers).

[Vbitz]

351f280 makes a image using the same rough logic as the alpine code. It should theoretically work but sadly the image it generates has too many layers.

It turns out that at least Podman and Docker are limited to 127 layers in a single image (is this a kernel limit?).

I guess I need to write some logic to merge all the layers into a single final layer. That should be easy enough.

[Vbitz]

598f50f merges all the layers together. They are now loadable in Docker.

Now for the installation scripts. RPM supports running scripts with Lua (https://rpm-software-management.github.io/rpm/manual/lua). A bunch of the scripts are written in Lua so I have to figure out how to execute them (and figure how to evaluate them in Starlark later).

I don't seem to have the metadata that indicates if a script is a shell script or a lua script. Let's get that from the metadata.

[Vbitz]

Other projects (like rpm-ostree) have run into the same problem from the looks of things.

The lua code is run by a interpreter embedded inside rpm. More modern versions of rpm offer a tool to run scripts called rpmlua but that was introduced Fedora 37 (https://docs.fedoraproject.org/cs/fedora/f37/release-notes/sysadmin/System_Utilities/).

I guess I have to figure out how rpm runs it's lua scripts to see if there's an alternative entry point. I feel like some of these scripts were introduced pretty recently so the version in Fedora 35 may not be that modern (The AFNI container uses Fedora 35).

[Vbitz]

The functionality is very old https://blog.labix.org/2004/03/23/embedding-lua-interpreter-into-rpm

A quick browse through the code suggests that the scripting functionality only has a few entry points and none that are externally visible.

As far as a way forward is concerned...

• One curious option would be to synthesize RPM files for those scripts and let RPM run the script. It's not going to be as easy as I hoped anyways.
• Another option is to write a parse_lua script which translates the Lua scripts into Starlark which can be handled internally or run by a simple bit of Go inside the final container. This library makes me thing that will be possible https://github.com/Shopify/go-lua

[Vbitz]

Turns out nix ran into the same multi-layer issue and came up with a great solution. https://grahamc.com/blog/nix-and-layered-docker-images/ I'll implement a version of the same thing by exposing the installation plan graph to Starlark.

I looked into the RPM Lua issue a little more and it seems only a few initial packages rely on it. I'll need to give more thought to what a solution looks like. rpm-ostree solves the problem by stubbing out the scripts with replacement shell scripts. I could probably do the same thing since I now know it's not that common.

I'm seeing a few smaller file system issues so I'll fix the tar conversion process first. Right now I'm using a hacky little Starlark function to populate the filesystem but I'll replace it with a real converter that takes a StarArchive (Eventually I'd like to replace that API as well but that's a story for another issue).

[Vbitz]

7f666aa removes make_fs and comments out the RPM scripts for now. They are still in the image but they won't execute when /.pkg/install.sh is called.

Next step is fixing the order of the NeuroDocker directives so I can start running shell scripts as well. I'll implement these as layers which get run in the /.pkg/install.sh script.

[Vbitz]

f589a4c fixes the order of the directives.

I still need to implement shell script handling. I want to test a bigger variety of the neurocontainer packages besides AFNI first.