cwl-test-to-galaxy-test-plan

---
type: research
tags:
  - target/galaxy
status: draft
created: 2026-05-03
revised: 2026-09-23
revision: 3
related_notes:
  - "[[iwc-workflow-testability-survey]]"
  - "[[iwc-test-data-conventions]]"
  - "[[planemo-asserts-idioms]]"
  - "[[iwc-shortcuts-anti-patterns]]"
  - "[[planemo-workflow-test-architecture]]"
  - "[[implement-galaxy-workflow-test]]"
  - "[[gxformat2-schema]]"
  - "[[gxformat2-workflow-inputs]]"
  - "[[galaxy-datatypes-conf]]"
summary: "Design guidance for Galaxy workflow inputs, outputs, and checkpoints that make IWC-style workflow tests possible."
---

# Design a Galaxy workflow that can be tested

Choose the workflow's public inputs and outputs while designing the analysis, before writing its `-tests.yml`. A workflow test supplies `job:` values by input name and checks `outputs:` by output name. It can only check results the workflow exposes. A good interface gives the test a reproducible input, an addressable result, and at least one assertion that would catch a meaningful wrong result.

This note concerns workflow design. [[iwc-test-data-conventions]] covers the test file's input fixtures, [[planemo-asserts-idioms]] covers assertion syntax, and [[iwc-shortcuts-anti-patterns]] explains when a smoke check is enough. [[iwc-workflow-testability-survey]] holds the broader corpus evidence.

## Name the public interface before writing tests

Give each input and promoted output a stable, descriptive public name. In an exported gxformat2 workflow this is commonly the entry's `id`. The structural [[gxformat2-schema]] also permits `label`. Check the **effective name in the workflow being tested** instead of assuming a step label, generated dataset name, or array position will become the test key. In the pinned [Scanpy workflow](https://github.com/galaxy
...

---
type: research
tags:
  - target/galaxy
status: draft
created: 2026-04-30
revised: 2026-09-23
revision: 3
related_notes:
  - "[[galaxy-workflow-testability-design]]"
  - "[[iwc-test-data-conventions]]"
  - "[[planemo-asserts-idioms]]"
  - "[[implement-galaxy-workflow-test]]"
  - "[[tests-format]]"
  - "[[iwc-conditionals-survey]]"
  - "[[iwc-map-over-lifecycle-survey]]"
  - "[[iwc-tabular-operations-survey]]"
  - "[[iwc-transformations-survey]]"
summary: "Judgments for when weak Galaxy workflow-test assertions are useful smoke checks and when they conceal untested results."
---

# IWC test-suite shortcuts and anti-patterns

A workflow test should fail when a meaningful result is wrong. Start by naming that failure, then choose an assertion that would catch it. A size, shape, or existence check can establish that an output was produced, but it cannot establish that the analysis is correct. Keep such a **smoke check** when the result is genuinely variable or opaque, and add a stable content check or workflow checkpoint whenever one is available. Using a weak check merely because another IWC test uses it is an anti-pattern.

These authoring judgments guide [[implement-galaxy-workflow-test]]. [[planemo-asserts-idioms]] covers assertion forms, [[iwc-test-data-conventions]] covers fixtures, and [[galaxy-workflow-testability-design]] covers which intermediate results to expose as workflow outputs.

## Choose by the failure the test must catch

| Output | Useful check | Anti-pattern |
| --- | --- | --- |
| Small, byte-stable text or data | Compare with an expected `file:`. A `checksum:` can cover a large byte-stable output. | Replacing an exact comparison with size alone when exact content is stable. |
| Variable JSON or HDF5 | Assert a stable property, group, key, or count 
...

---
type: research
tags:
  - target/galaxy
status: draft
created: 2026-04-30
revised: 2026-09-23
revision: 4
related_notes:
  - "[[galaxy-workflow-testability-design]]"
  - "[[iwc-shortcuts-anti-patterns]]"
  - "[[planemo-asserts-idioms]]"
  - "[[implement-galaxy-workflow-test]]"
  - "[[tests-format]]"
  - "[[iwc-tabular-operations-survey]]"
summary: "IWC and Planemo workflow-test fixture shapes, file provenance, collection identifiers, input hashes, and built-in index values."
---

# IWC test data conventions

An IWC workflow test pairs each workflow with a sibling `<workflow>-tests.yml`. Each YAML test case supplies `job:` inputs under the workflow's input labels and `outputs:` assertions under its output labels. The optional `test-data/` directory holds local fixtures. These are the [IWC contribution layout](https://github.com/galaxyproject/iwc/blob/main/workflows/README.md) and [Planemo test-format](https://planemo.readthedocs.io/en/latest/test_format.html) contracts. For decisions about the workflow interface before writing fixtures, see [[galaxy-workflow-testability-design]]. For the exact supported YAML vocabulary and static validation, see [[tests-format]].

Examples below come from the cited IWC files or Planemo documentation. Corpus observations describe the workflows sampled for this note, not a rule that every IWC workflow follows.

## Fixture locations and provenance

A `class: File` input can use `path:` for a local fixture or `location:` for a URI. Planemo resolves a relative `path:` from the workflow and test directory. IWC's contribution guide recommends small inputs and suggests publishing a toy dataset to Zenodo for a permanent URL. The repository also permits an optional local `test-data/` directory. It gives no universal size cutoff or mandatory hos
...

---
type: research
tags:
  - target/galaxy
status: draft
created: 2026-04-30
revised: 2026-09-23
revision: 7
related_notes:
  - "[[galaxy-workflow-testability-design]]"
  - "[[iwc-test-data-conventions]]"
  - "[[iwc-shortcuts-anti-patterns]]"
  - "[[implement-galaxy-workflow-test]]"
  - "[[tests-format]]"
  - "[[planemo-workflow-test-architecture]]"
  - "[[validate-tests]]"
  - "[[iwc-tabular-operations-survey]]"
  - "[[galaxy-discover-datasets]]"
summary: "Choose Galaxy workflow-test output assertions by the failure they should catch, the output's stability, and the available checkpoints."
---

# Choosing Planemo workflow-test assertions

Name the wrong result a test must catch before choosing YAML syntax. For a fixed input and byte-stable output, compare the result with an expected file or checksum. For a variable result, assert the stable properties that would fail if the analysis were wrong. A size, shape, or existence check is a useful smoke test when content is genuinely variable, ideally alongside a stable checkpoint. It is an anti-pattern when it replaces a more meaningful check that is available. [[iwc-shortcuts-anti-patterns]] develops that judgment, while [[iwc-test-data-conventions]] covers input fixtures and [[tests-format]] supplies the exact assertion vocabulary.

## 1. Choose by the property, then the output type

| Output | Meaningful check to try | If the result varies |
| --- | --- | --- |
| Text report or HTML | [[tests-format#has_text_model|has_text]] for a stable result, section, or value. Use [[tests-format#has_line_model|has_line]] for a complete stable line. | Use [[tests-format#has_text_matching_model|has_text_matching]] or [[tests-format#has_line_matching_model|has_line_matching]] for a bounded variable field. Line count or size can supplement
...

---
type: research
title: "Planemo workflow-test architecture"
tags:
  - tool/planemo
  - target/galaxy
status: draft
created: 2026-05-02
revised: 2026-05-11
revision: 3
related_notes:
  - "[[galaxy-workflow-testability-design]]"
  - "[[galaxy-tool-job-failure-reference]]"
  - "[[galaxy-workflow-invocation-failure-reference]]"
  - "[[planemo-asserts-idioms]]"
related_molds:
  - "[[run-workflow-test]]"
  - "[[debug-galaxy-workflow-output]]"
  - "[[implement-galaxy-workflow-test]]"
sources:
  - "~/projects/repositories/planemo/planemo/commands/cmd_test.py"
  - "~/projects/repositories/planemo/planemo/commands/cmd_run.py"
  - "~/projects/repositories/planemo/planemo/galaxy/activity.py"
  - "~/projects/repositories/planemo/planemo/galaxy/invocations"
  - "~/projects/repositories/planemo/planemo/galaxy/config.py"
summary: "Reference for Planemo workflow test/run architecture, Galaxy modes, API polling, and noisy failure boundaries."
---

# Planemo Workflow-Test Architecture

This note describes Planemo architecture relevant to workflow tests and workflow runs. It is reference material for Molds that need to run tests or interpret Planemo artifacts, not a command-selection recipe.

## Main Commands

| User action | Command | Core behavior |
|---|---|---|
| Full workflow test | `planemo test <workflow>` ([[planemo-test]]) | Finds test definitions, starts or targets Galaxy, stages inputs, invokes workflow, checks assertions, writes reports. |
| Direct run | `planemo run <workflow> <job.yml>` | Runs one workflow/job pair and can download outputs without assertion checks. |
| Recheck assertions | `planemo workflow_test_on_invocation <tests.yml> <invocation_id>` ([[planemo-workflow_test_on_invocation]]) | Runs test assertions against an existing invocation without rerunning the wo
...