Planemo CI action

Installs planemo, discovers changed workflows and tools, and allows to lint, test or deploy them.

The reference use cases of this action are the pull request and continuous integration workflows of the intergalactic utility comission (IUC).

The action runs in one of six modes which are controled with the mode input. Possible values are:

• setup: This is the default.
  • Optionally do a fake planemo test run to fill .cache/pip and .planemo for caching.
  • Determine the relevant commit range.
  • Determine the set of relevant repositories and tools.
  • Determine the number of chunks and the chunk list.
• lint: Lint tools with planemo shed_lint (resp. workflows with planemo workflow_lint) and check presence of repository metadata files (.shed.yml).
• test: Test tools or workflows with planemo test.
• combine: Combine the outputs from individual tool tests (planemo merge_test_reports) and create html/markdown reports (planemo test_reports).
• check: Check if any of the tool tests failed.
• deploy: Deploy tools to a toolshed using planemo shed_update and workflows to a github namespace, resp.

If none of these modes is set then a setup mode runs.

In all modes required software will be installed automatically, i.e. planemo and jq. The version of planemo can be controlled with the input planemo-version (default "planemo", i.e. the latest version).

Assumptions

The action currently only works on github actions workflows using an Ubuntu image.

Assumptions on the repository

Two files .tt_skip and .tt_biocontainer_skip containing paths (or prefixes of paths) can be used to skip or modify the testing for tools.

Tools/workflows in a path that has a prefix in:

• .tt_skip are ignored in all modes
• .tt_biocontainer_skip are not tested using containers but conda is used for resolving requirements.

Tools and workflows are discovered in all directories, except for packages/ and deprecated/. These directories may be absent.

A global .lint_skip file and per repo .lint_skip files can be used to list tool/workflow linters that should be skipped.

For workflow repositories a .wt_instance file in a workflow folder contains the instance to run the workflow tests against (note that the instance must not contain the https:// prefix). In order to be able to use this, an API key for the instance used in the repo needs to be passed via galaxy-user-key. If multiple instance are used in the repository (each .wt_instance can contain only one instance, but multiple ones may be used for the whole repo) the secret can be json formatted as follows:

{
    "usegalaxy.eu": "EU_API_KEY",
    "usegalaxy.org": "ORG_API_KEY"
}

In order to be usable in pull requests from forks the secret should be stored as environment secrets and the test step should be assigned this environment, i.e. the test step will require approval.

Setup mode

This mode runs if no other mode is selected. It:

• runs planemo test on a mock tool (if create-cache is set to true)

• determines the relevant set of tool/workflow repositories and tools with planemo ci_find_repos and planemo ci_find_tools, respectively.

  • for push and pull_request events (using GITHUB_EVENT_NAME) tools and repositories that changed in the commit range
  • all tools and repositories otherwise

• calulates the number of chunks to use for tool testing and the list of chunks

Optional inputs:

• workflows: look for workflows instead of tools
• create-cache (default false)
• galaxy-branch (default latest Galaxy release)
• galaxy-source (default galaxyproject)
• max-chunks (default 20)
• python-version (default "3.11")

Outputs:

• commit-range: The used commit range.
• tool-list: List of tools (empty if workflows is true).
• repository-list List of repositories.
• chunk-count: Number of chunks to use.
• chunk-list: List of chunks

Lint mode

Calls planemo shed_lint for each repository and checks if each tool is in a repository (i.e. metadata like .shed.yml is present).

Required inputs:

• repository-list
• tool-list

Optional inputs:

• report_level: all|warn|error (default: all)
• fail_level: warn|error (default: error)
• additional-planemo-options: additional options passed to planemo lint. Here this is used to overwrite the warn level of planemo lint.
• galaxy-branch: Galaxy branch to test against (default: master). Used for Galaxy package installation when testing with non-master/main branches.
• galaxy-fork: Galaxy fork to test against (default: galaxyproject). Used for Galaxy package installation when testing with non-master/main branches.

Output:

• creates a file lint_report.txt

Test mode

Runs planemo test for each tool in a chunk using ci_find_tools. Note that none of the tests will produce a non-zero exit code even if the tests fail. Success needs to be checked with the check mode after combining the outputs of the chunks.

Required inputs:

• repository-list: List of repositories
• workflows: test workflows
• setup-cvmfs: setup CVMFS (only useful for testing workflows)
• chunk: Current chunk
• chunk-count: Maximum chunk

Optional inputs:

• database_connection
• galaxy-branch
• galaxy-source
• python-version
• additional-planemo-options: additional options passed to planemo test, see for instance here
• galaxy-slots: number of slots (threads) to use in Galaxy jobs (sets the GALAXY_SLOTS environment variable)
• test_timeout: Maximum runtime of a single test in seconds, default: 86400
• galaxy-user-key: API key(s) used for testing agains online instances (note: use secrets for this). See "Assumptions on the repository".
• previous-run-id: re-run only previously-failed tests (see "Retesting only previously-failed tests" below).
• github-token: GitHub token used to download the previous run's artifacts (only needed together with previous-run-id).

Output:

The test mode creates a directory upload/ containing the test results as json file.

Retesting only previously-failed tests

Set previous-run-id to the run ID of an earlier run to re-test only the test cases that failed in that run instead of running the full test suite. When set, test mode:

• downloads that run's Tool test output {chunk} artifact via gh run download (the gh CLI is preinstalled on GitHub-hosted runners; on self-hosted runners it must be installed. A valid github-token must be passed);
• restricts the run to the previously-failed test cases via planemo test --failed --failed_json, so only failed tests are submitted to the engine.

The caller must pass the same repository-list, chunk-count, and chunk as the original run so that the identical tool groups are reconstructed. This requires planemo ≥ the release that ships the --failed/--failed_json flags (see galaxyproject/planemo#1653).

The output is identical to a regular test run (a tool_test_output.json/.html uploaded as Tool test output {chunk}), so the combine and check modes work unchanged downstream.

Example workflow_dispatch job:

on:
  workflow_dispatch:
    inputs:
      rerun-run-id:
        description: 'Run ID of the previous run to re-test failures from'
        required: true
        type: string

jobs:
  retest:
    strategy:
      fail-fast: false
      matrix:
        chunk: ${{ fromJson(needs.setup.outputs.chunk-list) }}
    steps:
      - uses: actions/checkout@v4
      - uses: galaxyproject/planemo-ci-action@main
        with:
          mode: test
          repository-list: ${{ needs.setup.outputs.repository-list }}
          chunk: ${{ matrix.chunk }}
          chunk-count: ${{ needs.setup.outputs.chunk-count }}
          previous-run-id: ${{ inputs.rerun-run-id }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/upload-artifact@v4
        with:
          name: 'Tool test output ${{ matrix.chunk }}'
          path: upload

Combine test outputs mode

Combines the test result of the chunked tests and create html or markdown reports.

Required input:

• json files need to be placed in a directory artifacts/.

Optional inputs:

• html-report (default: false)
• markdown-report (default: false)

Output:

• statistics: (text) historam of the number of tests that passed, errored, failed, skipped.

A directory upload/ containing the combined test results as json file and optionally as html/markdow files (named tool_test_output.[json|html|md]).

Test combined outputs mode

Check the combined outputs for failed test runs. If a failed test is found exit code 1 is returned.

Required input:

tool test results in upload/tool_test_output.json

Output:

statistics: Text containg the number of successful and failed tests

Deploy mode

Deploy all repositories to a toolshed.

Required inputs:

• workflows: deploy workflows to github namespace
• workflow-namespace
• shed-target toolshed name (e.g. "toolshed" or "testtoolshed")
• shed-key API key for the toolshed