Click2CWL - from a Click context to a CWL document

Rational and context

EO application developers use Click to create command line tools to process EO data.

Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary. It’s the “Command Line Interface Creation Kit”. It’s highly configurable but comes with sensible defaults out of the box. It aims to make the process of writing command line tools quick and fun while also preventing any frustration caused by the inability to implement an intended CLI API.

To deploy these applications on an Exploitation Plaform on the Cloud, the EO application developers generate a CWL document that becomes an Application Package.

The repo contains a helper Python module that exploits the information contained in a Click context object to generate the CWL document and eventually the parameters.

This helper Python module can thus be used to add "CWL dump" behaviour to a Python CLI that uses Click by adding an import and call to the dump(ctx) function provided in the helper Python module.

Installation

Install this Python module in your conda environment using conda or mamba (see this article to learn about mamba):

pip install -c terradue click2cwl

Note: Add click2cwl dependency and terradue channel in your environment.yml file, e.g.:

name: my_env
channels:
  - terradue
  - conda-forge
dependencies:
  - python
  pip:
  - click2cwl

Usage

First import the click2cwl.dump function with:

from click2cwl import dump

Then, in the Python application entry point function, update and enrich the Click function decorator with the information for generating the CWL document.

The information defined at the Click function decorator is explained below:

@click.command

The @click.command decorator must:

• set the allow_extra_args flag to True. This allows using the command line interface to pass the runtime information to generate the CWL.
• set the short_help to define the CWL Workflow class label
• set the help to define the CWL Workflow class doc

@click.command(
    short_help="This is Workflow class label",
    help="This is Workflow class doc",
    context_settings=dict(
        ignore_unknown_options=True,
        allow_extra_args=True,
    ),
)

@click.option

The @click.option decorator must:

• define the option type that can be click.Path for a Directory, a click.File for a File or the default, a String to map as string
• set the help to define the CWL Workflow class doc and label
• set the required flag to False if the parameter is optional

@click.option(
    "--input_path",
    "-i",
    "input_path",
    type=click.Path(),
    help="this input path",
    multiple=False,
    required=True,
)
def cli(): ...

@click.argument

The @click.argument decorator must:

• define the argument type (same types as for @click.option)
• must OMIT the help (this is a limitation of click, see pallets/click#587)
• it will be automatically marked as required=True

@click.argument(
    "input_path",
    type=click.Path(),
)
def cli(): ...

The advantage of using @click.argument over @click.option is that it will correspond to a positional argument, for cases where to option flag applies (i.e.: --<some-option>). However, it comes with limitations in terms of additional parameters offered by click.

@click2cwl.dump

Finally, invoke the dump function in your Click decorated function with:

@click.command ...
@click.option ...
@click.pass_context
def main(ctx, **kwargs):

    click2cwl.dump(ctx)

    print("business as usual")
    print(kwargs)

if __name__ == '__main__':
    main()

Note: See the examples folder to discover typical use cases.

CWL Exporters

There are multiple exporters to deal with the various contents:

• click2cwl.clt.CLTExport: CWL CommandLineTool directly
• click2cwl.cwlexport.CWLExport: CWL Workflows and its underlying CommandLineTool
• click2cwl.paramexport.ParamExport: CWL Job Parameters

These can be invoked directly instead of the @click2cwl.dump operation to extract the contents separately from the CLI invocation.

Building the application

Build your application with:

hatch build

When the app help is invoked, it will only show the arguments defined by the click.decorators:

myapp --help

but it now supports additional args to drive the generation of the CWL document and associated parameters:

The additional args are:

• --dump cwl|params|clt. Example --dump cwl --dump params will dump the CWL document and the CWL parameters template in YAML. clt will dump the CWl CommandLineTool class only (no Workflow)
• --requirement with requirement=value where requirement here is one of "coresMin", "coresMax", "ramMin", "ramMax". Example: --requirement ramMax=1 --requirement ramMin=2
• --docker <docker image> if set, the DockerRequirement hint is set to pull the <docker image>
• --env sets environment variables in the CWL with env_var=env_var_value. Example --env A=1 --env B=2 where A and B are the environment variables to set in the CWL EnvVarRequirement requirement
• --wall-time <wall-time in seconds> if set, the ToolTimeLimit cWL requirement is set and the CWL document version is set to v1.1 (instead of default 1.0)
• --cwl-version, if set, the cwlVersion uses that value

Try me on Binder