[FEATURE] Make currentness of used input files configurable

[magnuspalmblad]

Feature Description

When tools are chained together that pass along multiple files, with at least one of the same type and format being generated by multiple tools, or provided as workflow input, there is currently no way to specify or require that the most recently generated file of that type and format is used. For example, mass spectrometry imaging workflows may pass along both an imzML metadata file (http://edamontology.org/format_3682) and ibd (http://edamontology.org/format_3839) and perform multiple operations (e.g. alignments, filtering) affecting one or both of these files.

Motivation

I am not requesting a feature that solves this issue for every use case (likely impossible), but in most cases, using the most recent version of the file(s) would make most sense, and it would be nice to see this reflected in the generated workflows presented to the user.

Proposed Solution

This issue may warrant further discussion, but possible solutions would be either making the necessary changes to APE itself, or by filtering the generated workflows afterwards, to keep only those that always use the most recent version of a file. This filter could be optional, but on by default.

APE implementation clarification

Currently, APE supports only hard constraints, and the user can in the configuration specify whether they want to enforce usage of ALL, at least ONE or NONE output per tool. See use_all_generated_data under runtime configuration. The problem occurs when we prefer a second output to be used.

Example

Here is an example of a workflow that generates Mass spectrometry data, imzML metadata file as part of the msiwarp step, but because it is a second output of the tool, it's usage is not enforced by APE.

Note:
I have manually updated the image with the second output, because the "taverna-style visualisation" does not present unused outputs. For that a user can request "data-flow visualisation" instead.

[zzzZZ-Zhao]

Attached example workflow diagram generated by the Workflomics platform, along with a screenshot of the tool annotation used in the process.

[eladrion]

Hi @magnuspalmblad , @zzzZZ-Zhao. The workflow looks like the most current ibd file is used. But is that done always? I'm not sure about that and would have to dig into APE

[magnuspalmblad]

For clarification, this is the bio.tools annotation of MSIwarp, right? So the expected workflow should use the imzML metadata file generated by MSIwarp, not the original workflow input.

[zzzZZ-Zhao]

Yes, exactly. the annotation is for MSIwarp. So ideally, the imzML file generated by MSIwarp should be used in the downstream steps, not the original input file. In fact, if a tool in the workflow has the same input and output format, the system will ultimately use the original input file, rather than the most recently generated one. So this is not an issue specific to MSIwarp — it's a common situation whenever there are repeated formats in the workflow.

[eladrion]

Okay, I see the problem.

[vedran-kasalica]

Hi all, I have updated the issue description to clarify the underlying problem. There are multiple ways to resolve this of course.

1. Ideally, we could add heuristics to the solver, to specify soft constraints, but that would result in slower runtime and would require time to implement.

2. Maybe a simpler solution would be to allow labelling tool outputs as primary and secondary and then to enforce the usage of all primary outputs by APE. I would not encourage using the constraint use all outputs (as it is implemented now) because many tools generate the same data in multiple formats (e.g., Comet), and we don't want to enforce the usage of all of them.

3. And finally (and maybe the easiest to apply to our current domains), we could add a hard constraint to the solver that says, "never re-use existing data if we have data of the same format in memory unused". This could be part of the run configuration, where the users can disable it if needed.

[magnuspalmblad]

Hi @magnuspalmblad , @zzzZZ-Zhao. The workflow looks like the most current ibd file is used. But is that done always? I'm not sure about that and would have to dig into APE

At least one of the two outputs must be used by some downstream component - in this case the next component. I assume the same thing could happen with the old ibd and the new imzML metadata file from MSIwarp being consumed by MALDIquant?

[magnuspalmblad]

Hi all, I have updated the issue description to clarify the underlying problem. There are multiple ways to resolve this of course.

1. Ideally, we could add heuristics to the solver, to specify soft constraints, but that would result in slower runtime and would require time to implement.
2. Maybe a simpler solution would be to allow labelling tool outputs as primary and secondary and then to enforce the usage of all primary outputs by APE. I would not encourage using the constraint use all outputs (as it is implemented now) because many tools generate the same data in multiple formats (e.g., Comet), and we don't want to enforce the usage of all of them.
3. And finally (and maybe the easiest to apply to our current domains), we could add a hard constraint to the solver that says, "never re-use existing data if we have data of the same format in memory unused". This could be part of the run configuration, where the users can disable it if needed.

Spontaneously, I would think 3 would be the best. I the user wants something else, it is likely a rather atypical workflow, and they would (presumably) be able to construct it themselves.