cwl-summary-to-galaxy-data-flow

---
type: research
subtype: component
title: "CWL workflow anatomy"
tags:
  - research/component
  - source/cwl
status: draft
created: 2026-05-10
revised: 2026-05-10
revision: 1
ai_generated: true
related_notes:
  - "[[summary-cwl]]"
  - "[[cwl-v1.2-schemas]]"
  - "[[galaxy-collection-semantics]]"
related_molds:
  - "[[summarize-cwl]]"
  - "[[cwl-summary-to-galaxy-interface]]"
  - "[[cwl-summary-to-galaxy-data-flow]]"
  - "[[cwl-summary-to-galaxy-template]]"
sources:
  - "https://www.commonwl.org/v1.2/Workflow.html"
  - "https://cwltool.readthedocs.io/en/stable/"
  - "https://github.com/common-workflow-language/cwl-utils#normalize-a-cwl-document"
  - "https://pypi.org/project/cwl-utils/"
  - "https://github.com/common-workflow-language/cwldep"
summary: "CWL structure relevant to summarize-cwl: normalized documents, steps, scatter, conditionals, requirements, and dependency handling."
---

# CWL Workflow Anatomy

CWL is a structured workflow language, not a pipeline framework that must be inferred from ecosystem conventions. The `summarize-cwl` Mold should therefore start from CWL's own validated object model and avoid recreating the heavy Nextflow extraction stack.

## Normalization Posture

Use `cwltool --validate` as the first gate. If validation fails, the summary should emit provenance plus validation diagnostics and stop before producing downstream-looking graph claims.

Use `cwl-normalizer` from `cwl-utils` as the default normalization surface. The cwl-utils README describes it as producing JSON CWL documents with dependencies packed together, upgrading to CWL v1.2 as needed, and optionally refactoring CWL expressions into separate steps. This is the right handoff for `summarize-cwl`: structured enough for extraction, still source-faithful, and not a Galaxy design
...

---
type: research
subtype: design-spec
title: "CWL when:/pickValue → Galaxy branching translation"
tags:
  - research/design-spec
  - source/cwl
  - target/galaxy
status: draft
created: 2026-05-11
revised: 2026-05-11
revision: 1
ai_generated: true
related_notes:
  - "[[cwl-pickvalue-to-galaxy]]"
  - "[[galaxy-paired-or-unpaired-collections]]"
  - "[[galaxy-collection-semantics]]"
  - "[[component-cwl-workflow-anatomy]]"
  - "[[galaxy-data-flow-draft-contract]]"
related_molds:
  - "[[cwl-summary-to-galaxy-interface]]"
  - "[[cwl-summary-to-galaxy-data-flow]]"
  - "[[cwl-summary-to-galaxy-template]]"
  - "[[compare-against-iwc-exemplar]]"
summary: "CWL `when:`/`pickValue` → Galaxy. Three honest translations (paired_or_unpaired input, native pick_value step, sibling workflows) plus how to pick among them."
---

# CWL `when:`/`pickValue` → Galaxy branching translation

Audience: a Mold author looking at a `summary-cwl.json` whose steps carry `when:` predicates and/or whose workflow outputs use `pickValue`, deciding which Galaxy translation to recommend.

## The three honest translations

CWL has two related branching mechanisms with no 1:1 gxformat2 equivalent (until galaxy#22222 — see `cwl-pickvalue-to-galaxy`):

- **`when:` on a step** — execute conditionally on a JS predicate.
- **`pickValue:` on a step input or workflow output** — fan in N candidate sources and pick `first_non_null` / `the_only_non_null` / `all_non_null`.

Three Galaxy-idiomatic translations are available; each is honest for a different source shape.

### Translation A — `paired_or_unpaired` collection (preferred when the discriminator is paired-vs-single)

When the CWL `when:` predicates discriminate the **paired-vs-single mode of read inputs** (the seqprep-subwf pattern: `single_reads: File?` trigger
...

---
type: research
subtype: design-spec
title: "Galaxy data-flow draft contract"
tags:
  - research/design-spec
  - target/galaxy
status: draft
created: 2026-05-02
revised: 2026-05-03
revision: 2
ai_generated: true
related_notes:
  - "[[nextflow-to-galaxy-channel-shape-mapping]]"
  - "[[nextflow-operators-to-galaxy-collection-recipes]]"
  - "[[galaxy-workflow-draft]]"
related_molds:
  - "[[nextflow-summary-to-galaxy-data-flow]]"
  - "[[cwl-summary-to-galaxy-data-flow]]"
  - "[[freeform-summary-to-galaxy-data-flow]]"
  - "[[nextflow-summary-to-galaxy-template]]"
  - "[[cwl-summary-to-galaxy-template]]"
  - "[[freeform-summary-to-galaxy-template]]"
  - "[[compare-against-iwc-exemplar]]"
  - "[[advance-galaxy-draft-step]]"
sources:
  - "https://github.com/galaxyproject/foundry/issues/54"
summary: "Defines the proposed boundary between Galaxy data-flow drafts, gxformat2 templates, and concrete step implementation."
---

# Galaxy Data-Flow Draft Contract

This is an architectural contract, not a schema. Evidence is strongest for Mold and Pipeline boundaries. Proposed fields are speculative until exercised by two or three worked translations.

## Boundary

The data-flow draft owns a target-shaped abstract DAG for Galaxy. It should not be valid `gxformat2` and should not resolve exact Tool Shed tools.

Data-flow draft owns:

- Galaxy-facing workflow inputs and outputs.
- Abstract nodes, edges, branches, collection mapping, collection reduction, and placeholder transformations.
- Input/output shape decisions such as `File`, `list`, `paired`, `list:paired`, or `list:list`.
- Conceptual Galaxy idioms: map-over, reduction, Apply Rules, collection cleanup, identifier synchronization, tabular bridge, fan-in / combine (concatenate, merge, or union N sources into one — a distinct node
...

---
type: research
subtype: design-spec
title: "Open-requirements ledger"
tags:
  - research/design-spec
  - target/galaxy
status: draft
created: 2026-06-16
revised: 2026-06-16
revision: 1
ai_generated: true
related_notes:
  - "[[galaxy-workflow-draft-format]]"
related_molds:
  - "[[advance-galaxy-draft-step]]"
  - "[[repair-galaxy-draft-topology]]"
  - "[[implement-galaxy-tool-step]]"
summary: "Carried unresolved-requirements artifact the source→Galaxy pipeline discharges or explicitly surrenders, autonomously."
---

# Open-requirements ledger

The `open-requirements-ledger` is a single artifact threaded through the source→Galaxy pipeline that records **obligations the pipeline has taken on but not yet met** — a declared output with no producer, a parameter whose value the source never pinned, a tool with no corpus exemplar. Each Mold that surfaces one **appends** it; each Mold whose decision closes one **marks it resolved**; the terminal path **surrenders** whatever remains open, explicitly, into the final artifact.

## Framing: obligations the pipeline discharges, not questions a human answers

This is deliberately *not* an "open questions for the user" list. The pipeline is autonomous — no human-in-the-loop gate is assumed. The ledger's consumers are **Molds and the loop's convergence gate**, with human readout a secondary affordance. An entry is closed by a downstream Mold doing work (wiring a producer, picking a wrapper, settling a value), or — when nothing can close it — surrendered: written into the final draft as a known, labelled gap rather than silently dropped or fabricated around.

The distinction matters because a "questions for a human" framing leaks an operator's personal interaction style into a tool meant to run inside anyone's harness. The ledger must
...

---
type: research
subtype: component
title: "CWL pickValue → Galaxy pick_value (post galaxy#22222)"
tags:
  - research/component
  - source/cwl
  - target/galaxy
status: draft
created: 2026-05-11
revised: 2026-05-11
revision: 1
ai_generated: true
related_notes:
  - "[[component-cwl-workflow-anatomy]]"
  - "[[galaxy-data-flow-draft-contract]]"
  - "[[galaxy-workflow-draft-format]]"
related_molds:
  - "[[cwl-summary-to-galaxy-data-flow]]"
  - "[[cwl-summary-to-galaxy-template]]"
summary: "CWL `pickValue` (first_non_null / the_only_non_null / all_non_null) → Galaxy's native `pick_value` workflow step added by galaxyproject/galaxy#22222."
---

# CWL `pickValue` → Galaxy `pick_value`

Audience: a Mold author who just saw a `pickValue:*` marker in a `summary-cwl.json` edge `via:` array (or a `WorkflowOutputParameter.output_source` multi-value carrying a `pickValue` hint) and needs to emit gxformat2.

## CWL `pickValue` — canonical semantics

Source: CWL v1.2 schema `Workflow.yml` (`PickValueMethod`) and the rendered spec at <https://www.commonwl.org/v1.2/Workflow.html#PickValueMethod>.

- **`first_non_null`** — "For the first level of a list input, pick the first non-null element. The result is a scalar. **It is an error if there is no non-null element.**"
- **`the_only_non_null`** — "For the first level of a list input, pick the single non-null element. The result is a scalar. **It is an error if there is more than one non-null element.**"
- **`all_non_null`** — "For the first level of a list input, pick all non-null values. **The result is a list, which may be empty.**"

Placement: declared on **both** `WorkflowStepInput` and `WorkflowOutputParameter` with identical semantics. Operates on the array produced when `source:` / `outputSource:` is multi-valued. First level only
...

---
type: research
subtype: component
title: "Galaxy collection semantics"
tags:
  - research/component
  - target/galaxy
status: draft
created: 2026-04-30
revised: 2026-05-05
revision: 3
ai_generated: false
license: MIT
license_file: LICENSES/galaxy.LICENSE
related_notes:
  - "[[galaxy-xsd]]"
  - "[[galaxy-collection-tools]]"
  - "[[galaxy-apply-rules-dsl]]"
  - "[[nextflow-to-galaxy-channel-shape-mapping]]"
  - "[[nextflow-operators-to-galaxy-collection-recipes]]"
  - "[[galaxy-tool-job-failure-reference]]"
  - "[[galaxy-workflow-invocation-failure-reference]]"
  - "[[iwc-transformations-survey]]"
  - "[[galaxy-discover-datasets]]"
sources:
  - "https://github.com/galaxyproject/galaxy/blob/7765fae934fbfdee77e3be5f5b235e43735273ae/lib/galaxy/model/dataset_collections/types/collection_semantics.yml"
companions:
  - "galaxy-collection-semantics.yml"
  - "galaxy-collection-semantics.upstream.myst"
summary: "Vendored formal spec of Galaxy dataset-collection mapping/reduction semantics, with labeled examples and pinned test references."
---

> **Vendored from upstream**, pinned at SHA `7765fae`. Two files live next to this note:
>
> - `galaxy-collection-semantics.yml` — the structured source. **Agents and casting should consume this.** It carries the `tests:` blocks that pin concrete Galaxy test names; the rendered upstream view drops them.
> - `galaxy-collection-semantics.upstream.myst` — Galaxy's auto-generated MyST/LaTeX rendering of the YAML, vendored only so the human view below has something to render. Sync is manual.
>
> **When to consult:** authoring or reasoning about Molds and patterns that touch `data_collection` inputs, map-over / reduction shape changes, sub-collection mapping, `paired_or_unpaired`, or `sample_sheet`.

```vendored-myst
file: galaxy-collection-s
...

# Collection Semantics

This document describes the semantics around working with Galaxy dataset collections.
In particular it describes how they operate within Galaxy tools and workflows.

:::{admonition} You Probably Don't Need to Read This
:class: caution

Any significantly sophisticated workflow language will have ways to collect data
into arrays or vectors or dictionaries and apply operations across this data (mapping)
or reduce the dimensionality of this data (reductions). Typically, this is explicitly
annotated with map functions or for loops. Galaxy however is designed to be a point
and click interface for connecting steps and running tools. It is important that steps
just connect and just do the most natural thing - and this is what Galaxy does.
This document just provides a mathematical formalism to that "what should just
intuitively work" that can be used to document test cases and help with implementation.
This is reference documentation not user documentation, Galaxy should just work.
:::

## Mapping

If a tool consumes a simple dataset parameter and produces a simple dataset parameter,
then any collection type may be "mapped over" the data input to that tool. The result of
that is the tool being applied to each element of the collection and "implicit collections"
being created from the outputs that are produced from those operations. Those implicit
collections have the same element identifiers in the same order as the input collection that is
mapped over. Each element of the implicit collections correspond to their own job and
Galaxy very naturally and intuitively parallelizes jobs without extra work from the user
and without any knowledge of the tool.


(BASIC_MAPPING_PAIRED)=
(BASIC_MAPPING_PAIRED_OR_UNPAIRED_PAIRED)=
(BASIC_MAPPING_PAIRED_OR_UNPAIRED_UN
...

- doc: |
    # Collection Semantics

    This document describes the semantics around working with Galaxy dataset collections.
    In particular it describes how they operate within Galaxy tools and workflows.

    :::{admonition} You Probably Don't Need to Read This
    :class: caution

    Any significantly sophisticated workflow language will have ways to collect data
    into arrays or vectors or dictionaries and apply operations across this data (mapping)
    or reduce the dimensionality of this data (reductions). Typically, this is explicitly
    annotated with map functions or for loops. Galaxy however is designed to be a point
    and click interface for connecting steps and running tools. It is important that steps
    just connect and just do the most natural thing - and this is what Galaxy does.
    This document just provides a mathematical formalism to that "what should just
    intuitively work" that can be used to document test cases and help with implementation.
    This is reference documentation not user documentation, Galaxy should just work.
    :::

    ## Mapping

    If a tool consumes a simple dataset parameter and produces a simple dataset parameter,
    then any collection type may be "mapped over" the data input to that tool. The result of
    that is the tool being applied to each element of the collection and "implicit collections"
    being created from the outputs that are produced from those operations. Those implicit
    collections have the same element identifiers in the same order as the input collection that is
    mapped over. Each element of the implicit collections correspond to their own job and
    Galaxy very naturally and intuitively parallelizes jobs without extra work from the user
    and without any knowledge of the tool.

...

---
type: research
subtype: component
title: "Galaxy paired_or_unpaired collection type"
tags:
  - research/component
  - target/galaxy
status: draft
created: 2026-05-11
revised: 2026-05-11
revision: 1
ai_generated: true
related_notes:
  - "[[galaxy-collection-semantics]]"
  - "[[component-cwl-workflow-anatomy]]"
related_molds:
  - "[[cwl-summary-to-galaxy-interface]]"
  - "[[cwl-summary-to-galaxy-data-flow]]"
  - "[[nextflow-summary-to-galaxy-interface]]"
summary: "Galaxy's `paired_or_unpaired` collection type: discriminated-union shape for paired-or-single reads, no workflow-level mode switch needed. Galaxy PR #19377."
---

# Galaxy `paired_or_unpaired` collections

Audience: a Mold author shaping a Galaxy workflow interface from an upstream (CWL / Nextflow / paper) source whose reads can be paired-end *or* single-end *or* a mixed batch of both.

## The shape

`paired_or_unpaired` is a Galaxy collection type modeling a **discriminated union of 1 or 2 elements**:

- **Unpaired variant** — one element with identifier `unpaired`.
- **Paired variant** — two elements with identifiers `forward` and `reverse`.

`list:paired_or_unpaired` lifts the same shape to a *heterogeneous* batch where some samples are paired and some are single-end — a representation that did not exist before this type. A `list:paired` forces every sample to be paired; a plain `list` of flat datasets loses pairing structure.

The type and rank `paired_or_unpaired` may occur at any rank within nested types (`list:paired_or_unpaired`, `list:list:paired_or_unpaired`) but **only at the deepest (innermost) rank** — the subtyping logic is implemented at the suffix level. See "Limitation: only deepest rank" below.

## When to reach for it (decision rule for translators)

Reach for `paired_or_unpaired` when the
...