run-workflow-test

Execute an assembled workflow’s test file via planemo and emit a structured pass/fail with the artifacts a debug pass needs. One invocation runs the test once, captures the evidence, and — on failure — classifies the failure modality and names the next reference surface to inspect. It does not repair anything; that is debug-galaxy-workflow-output‘s job.

Sequence

0. Establish the test definition. Locate the workflow’s test file before anything else. If none can be found, do not abort: emit workflow-test-result.json with status: test-definition-missing, the paths searched, and the workflow it was searched for, then stop. Emit status: not-run plus a not_run_reason instead when a test exists but cannot be executed — a static validate-tests failure, missing test data, an unavailable Galaxy, an uninstallable tool, a timeout, or a checkout the caller has not confirmed as trusted. An environment that prevented the attempt is not-run, including an import refused for a missing permission or an uninstallable tool. A workflow Galaxy refused for its own content or through a server defect is fail (step 5). A downstream consumer needs to know that nothing ran; it must never have to infer it from a missing file.
1. Validate before running. When a test file is present, run validate-tests for the static schema and workflow-label checks first. A run is expensive; do not spend one on a test file that fails static validation.
2. Pick the Galaxy mode. Run against a Planemo-managed Galaxy or an existing/external Galaxy. Planemo-managed is the default and needs no pre-provisioned server: planemo test bootstraps its own Galaxy and installs the workflow’s tools from the Tool Shed/conda — so the absence of a running Galaxy is not a reason to skip the run. Use an existing/external Galaxy only when you deliberately want to target one (shared instance, pre-installed heavy tools/reference data, specific credentials). The real cost of the managed path is install/runtime weight (large tools or multi-GB reference databases), which is a deliberate deferral, not an impossibility. Record which mode was used, how tools, workflows, and test data were staged, and the URLs or API credentials a follow-up inspection would need. The choice and its consequences are guided by planemo-workflow-test-architecture.
3. Do not pin a Galaxy version. Leave --galaxy_branch off the command unless the user or the harness supplied a specific branch, and never guess a release_* value — Planemo’s default targets the newest Galaxy, which is the only version guaranteed to understand every construct a freshly authored workflow can contain. If a branch was supplied, record it in the output so a later failure can be read against it.
4. Run and capture. Drive planemo test with structured output enabled. Preserve the invocation id, history id, workflow id, the Planemo structured result, and any test-output artifact paths — these are the inputs the debug Mold consumes. When Galaxy rejects the workflow import, none of these exist: Planemo stops before writing structured output (planemo-workflow-test-architecture). Record the ids as absent and preserve the Planemo terminal log, the Galaxy server log, and the import request’s HTTP status and response body instead.
5. Classify on failure. When the run exits non-zero or reports failed assertions, failed jobs, a failed invocation, missing outputs, or upload/staging problems, identify the observed failure modality and the single next reference surface to open: Planemo result, Galaxy job API, Galaxy invocation API, history contents, the test assertion report, the workflow import response, or the Galaxy server log. A workflow that Galaxy refused to import for its content or through a server defect is modality workflow-import-rejected with status: fail, not not-run—Galaxy evaluated the workflow itself, so the response and the server log are runtime evidence about it. Name the import response as the next surface for a 4xx that carries a message, and the server log for a 500. Use planemo-asserts-idioms to read assertion failures and galaxy-workflow-invocation-failure-reference to preserve invocation identifiers and state.
6. Hand off. Emit the structured summary carrying a status from the closed set pass | fail | test-definition-missing | not-run — green, or red with modality + captured artifacts + the named next surface — for debug-galaxy-workflow-output. A test-definition-missing or not-run result is never reported as a pass and never as a Planemo failure: neither carries a failure modality or a next reference surface, because nothing ran. A consumer must read those two states as no runtime evidence, not as a debug target.

Keep this Mold’s output a faithful record of what happened, not a diagnosis. Mislabeling a staging failure as an assertion failure here sends the debug pass to the wrong reference surface.