N4 Preprocessing high memory usage

[levje]

Describe the bug
When running the sf-tractomics pipeline on CamCAN (when running on MyeloInferno everything was working fine), I had the N4_PREPROC require more than 16 Gb of RAM. As of now, the process is labeled as "high_memory" which attributes 16 Gb of RAM to it. However, since the processes were being killed on 16Gb of RAM they were automatically re-launched on 32 Gb RAM (which crashed because I have 31,98 Gb of RAM on my local desktop). It also is annoying to have my processes being launched twice because of higher RAM usage than expected.

Discussed with @AlexVCaron in scilus/sf-tractomics#71 and was asked to tag @gdevenyi to know if you had an idea if this is normal.

To Reproduce
Run sf-tractomics on CamCAN data.

Expected behavior
Either the memory requirements should be slightly increased, or there's a RAM usage problem to fix.

[gagnonanthony]

What's the resolution of the CamCAN data? I vaguely remember that on a high-resolution dataset (0.5mm), the N4 process took a long time, which I assume was also memory-intensive.

[gdevenyi]

If the N4 module has a fixed SHRINK (-s) parameter then high-resolution data will have resolution_ratio^3 more memory usage (and compute time) for higher res data compared to regular data.

A fix to the module would be to use the resolution of the data to compute a shrink which sets a fixed resolution for estimating the bias field. In my preprocessing pipelines I fix human-sized data at 4mm sampling for N4 runs.

[levje]

The (CamCAN) data I ran had a T1w of 192x256x256 with a resolution of 1mm isotropic and a DWI of 96x96x66 with resolution of 2mm isotropic.

[gagnonanthony]

I think the parameters for N4 are wrongly set in sf-tractomics. In the config file, this is the parameters supplied:

withName: ".*:TRACTOFLOW:PREPROC_DWI:N4_DWI" {
    ext.bspline_knot_per_voxel  = 0.25
    ext.shrink_factor           = 2
}

While in the nf-neuro documentation for the N4 module, we set the shrink_factor to 4 (and bspline_knot_per_voxel is no longer an argument). It seems to be a residue of the first implementation of nf-tractoflow ? @AlexVCaron @levje

[gdevenyi]

The (CamCAN) data I ran had a T1w of 192x256x256 with a resolution of 1mm isotropic and a DWI of 96x96x66 with resolution of 2mm isotropic.

Can you share any logs? Even memory usage > 16GB seems way way higher than I would ever see in N4.

[AlexVCaron]

I think the parameters for N4 are wrongly set in sf-tractomics. In the config file, this is the parameters supplied:

withName: ".*:TRACTOFLOW:PREPROC_DWI:N4_DWI" {
    ext.bspline_knot_per_voxel  = 0.25
    ext.shrink_factor           = 2
}

While in the nf-neuro documentation for the N4 module, we set the shrink_factor to 4 (and bspline_knot_per_voxel is no longer an argument). It seems to be a residue of the first implementation of nf-tractoflow ? @AlexVCaron @levje

@gagnonanthony you're right, but then only shrink shifts by a factor of 2. It does mean 8 times more memory space for operations though, so it might be the culprit.

[AlexVCaron]

@levje can you output some logs for Gab ? Edit the module to get as much verbosity as you can, I'm not sure we allow configuration for this.

[AlexVCaron]

@levje @gdevenyi Any updates on this ?

[gdevenyi]

Still waiting for some more details.

[levje]

I just reran a single subject of my data and I ran my pipeline on rorqual and wasn't able to reproduce the error for some reason. In total, I ran N4 on over 1400 images, and it didn't seem to cause any RAM issues problem. I think we can either close this issue, or I can try to make a more thorough investigation in a couple days if needed.

[AlexVCaron]

Please scrub a bit and find what changed. Either the container, the N4 configuration or the resource allocation changed. What version of Nextflow are you using ? What did the trick here ?

[gdevenyi]

Either way, I think you should consider moving from a fixed shrink factor to a fixed resolution for performing N4, as this will make the outputs more stable/reproducible across multiple datasets