Foundry
Home Schema

Galaxy workflow test plan

JSON Schema for the intermediate Galaxy workflow test-plan handoff produced by the test-plan Molds and consumed by implement-galaxy-workflow-test.

draft schematarget/galaxy
Revised
2026-06-16
Rev
1
Related
schema galaxy-workflow-test-plan @galaxy-foundry/foundry @ 0.0.0 upstream ↗

Galaxy Workflow Test Plan

Intermediate, reviewable Galaxy workflow test-plan handoff produced by a *-test-to-galaxy-test-plan Mold (nextflow, cwl, or freeform) and consumed by implement-galaxy-workflow-test. It preserves test intent, fixture provenance, assertion intent, tolerances, label assumptions, unresolved mappings, and intentional omissions before any concrete tests-format `*-tests.yml` is authored. It is NOT the final test artifact: assertion vocabulary is referenced by family name from the tests-format schema, not duplicated here.

12 sections. Anchor links per section (e.g. #root) are stable.

Galaxy Workflow Test Plan

root

Intermediate, reviewable Galaxy workflow test-plan handoff produced by a *-test-to-galaxy-test-plan Mold (nextflow, cwl, or freeform) and consumed by implement-galaxy-workflow-test. It preserves test intent, fixture provenance, assertion intent, tolerances, label assumptions, unresolved mappings, and intentional omissions before any concrete tests-format `*-tests.yml` is authored. It is NOT the final test artifact: assertion vocabulary is referenced by family name from the tests-format schema, not duplicated here.

field type req description
omissions → Omission[] ✓ Outputs or behaviors deliberately left unasserted, with rationale, so the gap is a recorded decision rather than an oversight.
plan_version string ✓ Test-plan schema major version.
source → SourceRecord ✓
test_cases → TestCase[] ✓ One entry per planned Galaxy workflow test (each becomes one tests-format test entry downstream).
unresolved → UnresolvedItem[] ✓ Mappings the plan could not resolve and that implement-galaxy-workflow-test (or a reviewer) must settle before authoring the final test file.
warnings → Warning[] ✓ Plan-construction warnings: source-test evidence that could not be translated, expression-derived shapes, missing fixtures, and similar.
workflow → WorkflowRef ✓

AssertionIntent

AssertionIntent
field type req description
confidence string ✓ How firmly the expected value and family are believed correct; synthesized assertions are typically medium or low.
element_identifier string | null ✓ Collection element identifier this assertion targets, for element-level (element_tests) assertions; null for top-level dataset assertions.
evidence string ✓ Whether this assertion was translated from upstream test evidence or synthesized from intent.
expected_value string | number | integer | boolean | null ✓ Concrete expected token, count, size, or comparison fixture when known; null when synthesized from intent without a concrete value.
family string ✓ tests-format assertion family or compare operator this intent maps to, by name, e.g. has_text, has_n_lines, has_n_columns, has_size, has_image_width, has_h5_keys, file, compare:diff, compare:sim_size. Names reference the tests-format schema vocabulary; this plan does not restate parameter shapes.
intent string ✓ What this assertion verifies, in plain language, so implement-galaxy-workflow-test can materialize the concrete assertion without re-reading the source tests.
tolerance → Tolerance | null ✓ Tolerance for the expected value when applicable; null for exact or existence-only assertions.

ExpectedOutput

ExpectedOutput
field type req description
assertion_intent → AssertionIntent[] ✓ Assertion intent for this output. May be empty when the output is intentionally unasserted (record the reason in omissions). Each entry names a tests-format assertion family by name rather than duplicating its full parameter object.
collection_shape string | null ✓ Galaxy collection shape for collection outputs, e.g. list, list:paired, list:list; null otherwise.
description string ✓ What this output is and why it is worth asserting.
label_status string ✓ resolved means the label matches a concrete workflow draft output; assumed means it is inferred from a brief; unresolved means the output label is not yet decided.
output_kind string | null ✓ Whether the output is a single dataset or a collection; null when undecided. Collection outputs carry element-level assertion intent via assertion_intent[].element_identifier.
workflow_label string | null ✓ Workflow output label to assert against; null when the output label is not yet known.

Fixture

Fixture
field type req description
checksum string | null ✓ Integrity hash when available, e.g. SHA-1:<hex>; null when unknown.
location string | null ✓ Concrete URL, repo-relative path, or data-table string when known; null when the fixture still needs resolution.
provenance string ✓ Where this fixture comes from or how it was chosen, e.g. a translated source test-data reference, a Zenodo record, or a note that test-data-resolution must supply it.
storage string | null ✓ Where the fixture data lives or should live: remote-url (Zenodo/EBI/SRA location), in-repo (committed test-data path), cvmfs-string (data-table/built-in-index bare string), generated-toy (to be synthesized), or unresolved (not yet sourced).

JobInput

JobInput
field type req description
collection_shape string | null ✓ Galaxy collection shape when the input is a collection, e.g. list, paired, list:paired, list:list; null for a single dataset or when unknown.
datatype string | null ✓ Galaxy datatype/extension hint, e.g. fastqsanger.gz, fasta, tabular; null when undecided.
description string ✓ What this input is and the role it plays in the test.
fixture → Fixture ✓
label_status string ✓ resolved means the label matches a concrete workflow draft input; assumed means it is inferred from a brief; unresolved means the binding label is not yet decided.
workflow_label string | null ✓ Workflow input label this job entry binds to; null when the label is not yet known.

Omission

Omission
field type req description
category string | null ✓ Coarse omission category; null when none fits.
reason string ✓ Why it is omitted, e.g. nondeterministic content, no stable checkpoint, or asserted only weakly elsewhere.
target string ✓ What is intentionally left unasserted, e.g. a workflow output label or a class of outputs.

SourceRecord

SourceRecord
field type req description
derived_from string ✓ Dominant basis for the plan: test-evidence means it was translated from existing upstream test fixtures/snapshots/job files (nf-test, CWL jobs); intent means it was synthesized from workflow intent and scenario-level expected outputs without upstream test evidence (freeform paper/interview sources).
kind string ✓ Source family of the Galaxy-targeting translator that produced this plan.
name string ✓ Human-oriented workflow or source name the plan was derived from.
notes string | null ✓ Free-text provenance about the source and how test evidence was (or was not) available.

TestCase

TestCase
field type req description
derived_from string ✓ Basis for this case: translated from upstream test evidence, synthesized from intent, or a mix.
doc string ✓ What this test exercises, in plain language; becomes the tests-format test `doc`.
expected_outputs → ExpectedOutput[] ✓ Planned workflow outputs to assert for this case.
id string ✓ Stable identifier for this planned test case.
job_inputs → JobInput[] ✓ Planned workflow job inputs for this case.
provenance string | null ✓ Where this case came from, e.g. an nf-test name, a CWL job file, a paper section, or an interview answer.

Tolerance

Tolerance
field type req description
kind string ✓ Tolerance operator: delta (absolute count/bytes, suffixes like 1K/1M allowed as strings), delta_frac (fraction such as 0.1), lines_diff (number of mutable header lines for compare:diff), or none.
magnitude string | number | integer | null ✓ Tolerance magnitude, e.g. 6, "10K", or 0.1; null when kind is none or the magnitude is undecided.
rationale string | null ✓ Why this magnitude was chosen, e.g. mutable VCF header line count, percentage of typical file size, or nondeterministic output.

UnresolvedItem

UnresolvedItem
field type req description
blocking boolean ✓ Whether implement-galaxy-workflow-test must resolve this before authoring a runnable test.
description string ✓ What could not be resolved and why.
kind string ✓ Category of the unresolved mapping.
suggested_resolution string | null ✓ A proposed way to resolve it when one is apparent; null otherwise.

Warning

Warning
field type req description
code string ✓ Stable warning code, e.g. untranslatable-snapshot, missing-fixture, expression-derived-shape.
message string ✓ Human-readable warning detail.
path string | null ✓ Pointer into the source or plan the warning refers to; null when not applicable.

WorkflowRef

WorkflowRef

How the plan relates to the Galaxy workflow it tests, and where its input/output labels came from.

field type req description
label_source string ✓ Provenance of the input/output labels this plan references: draft means they were read from a concrete gxformat2 workflow draft (verifiable); interface-brief means they came from an upstream Galaxy interface brief; assumed means they were inferred and are unconfirmed against any workflow.
notes string | null ✓ Free-text notes about label confidence and workflow-binding assumptions.
title string | null ✓ Galaxy workflow title/name when known.

This page is auto-rendered from the JSON Schema authored in this repo (orphan schema — no TypeScript producer Mold owns it, like summary-cwl and galaxy-tool-summary). Each $def becomes a section below with a stable anchor ID — Mold bodies and research notes can deep-link individual shapes via [[galaxy-workflow-test-plan#AssertionIntent]].

The on-disk artifact is YAML, so it stays a reviewable handoff while remaining machine-validatable; YAML is a JSON superset, so the same AJV schema gates both.

Source-of-truth chain:

  1. packages/foundry/src/schemas/galaxy-workflow-test-plan/galaxy-workflow-test-plan.schema.json — canonical JSON, hand-edited as part of the Mold/cast loop.
  2. packages/foundry/scripts/sync-schema.mjs regenerates the typed galaxy-workflow-test-plan.schema.generated.ts const wrapper at prebuild.
  3. Published as part of @galaxy-foundry/foundry, exporting galaxyWorkflowTestPlanSchema and the foundry validate-galaxy-workflow-test-plan subcommand.

Generated skills should validate emitted plans with:

foundry validate-galaxy-workflow-test-plan galaxy-test-plan.yml

What This Models — and What It Does Not

The test plan is the intermediate handoff between a source-specific test-plan producer (nextflow-test-to-galaxy-test-plan, cwl-test-to-galaxy-test-plan, freeform-summary-to-galaxy-test-plan) and implement-galaxy-workflow-test, which authors the final *-tests.yml. It preserves the intent and provenance of a test before any concrete tests-format file exists:

  • test_cases[] — one entry per planned Galaxy workflow test, each with job_inputs and expected_outputs.
  • job_inputs[] — workflow-label binding (workflow_label + label_status), collection_shape, datatype, and a fixture with storage class, location, checksum, and provenance.
  • expected_outputs[] — output-label binding, output_kind, collection_shape, and assertion_intent[].
  • assertion_intent[] — a tests-format assertion family by name (has_text, has_n_lines, has_size, compare:diff, …), a plain-language intent, an optional expected_value, an optional tolerance, an optional collection element_identifier, plus evidence and confidence.
  • unresolved[] / omissions[] / warnings[] — mappings left open, outputs deliberately unasserted (with rationale), and plan-construction warnings.

Intentional Non-Goals

  • No duplication of tests-format. Assertion families are referenced by name; their parameter shapes, defaults, and discriminators live in the tests-format schema. The plan records which family and why, not the final YAML.
  • Not the final artifact. implement-galaxy-workflow-test converts a schema-valid plan into a tests-format *-tests.yml and runs the static + workflow-label gates; the plan never substitutes for that file.
  • Source-agnostic structure. The schema does not encode nf-test, CWL, or paper-specific structure; source.kind is a provenance label only. The derived_from / evidence axis (test-evidence vs intent) distinguishes a plan translated from upstream test fixtures from one synthesized from workflow intent — the only difference a freeform source needs. That axis appears at three nested levels by design: source.derived_from is the plan’s dominant basis, test_cases[].derived_from lets a mostly-synthesized plan still carry an individual translated case (or mixed), and each assertion_intent[].evidence records the basis of one assertion — they layer, they are not interchangeable.

Expressing Unknowns

A freeform-sourced plan is synthesized from intent and carries pervasive uncertainty; a nextflow/CWL plan is translated from concrete test evidence. The schema makes uncertainty explicit rather than absent:

  • Nullable fields (workflow_label, datatype, collection_shape, expected_value, location, checksum, …) record “unknown” as null, not as an omitted key.
  • label_status (resolved / assumed / unresolved) and fixture.storage: unresolved flag bindings and fixtures that are not yet pinned.
  • confidence (high / medium / low) on each assertion intent lets synthesized assertions declare low confidence without dropping the assertion.
  • unresolved[] carries the mappings a reviewer or implement-galaxy-workflow-test must settle, with a blocking flag.

Incoming References (11)