pipeline-nextflow-to-cwl

SKILL.md

# pipeline-nextflow-to-cwl

Harness for the **NEXTFLOW → CWL** Foundry pipeline. Runs the constituent skills in order inside a single per-run working directory. Assembled from `content/pipelines/nextflow-to-cwl.md` (revision 2) — regenerate with `foundry-build assemble-pipeline nextflow-to-cwl` if the pipeline changes; do not hand-edit.

Most of this pipeline's Molds are not yet cast, so this harness is mostly manual checkpoints today; it still fixes the phase order, the per-run working directory, and the few real casts. It strengthens as the remaining Molds are cast.

## When To Use

- Direct path from a Nextflow pipeline to a CWL Workflow + CommandLineTool set.

## Bootstrap (install these CLIs first)

Install the harness CLIs every constituent skill invokes before driving the pipeline. Deduped across all phases; bioinformatics tools the constructed workflow installs are out of scope (the discovery phase pins those).

- **`foundry`** (foundry). `npm install -g @galaxy-foundry/foundry`.
  Ephemeral run: `npx --package @galaxy-foundry/foundry foundry`.
  Check: `foundry --help`.
  Docs: https://github.com/galaxyproject/foundry/blob/main/packages/foundry/README.md
- **`gxwf`** (gxwf). `npm install -g @galaxy-tool-util/cli`.
  Ephemeral run: `npx --package @galaxy-tool-util/cli gxwf`.
  Check: `gxwf --version`.
  Docs: https://github.com/jmchilton/galaxy-tool-util-ts/tree/main/packages/cli
- **`planemo`** (planemo). `uv tool install planemo==git+https://github.com/jmchilton/planemo@a9b8b8bc7ab3b12035d53bdb5383fe450413d9f3` (or `pip install planemo==git+https://github.com/jmchilton/planemo@a9b8b8bc7ab3b12035d53bdb5383fe450413d9f3`).
  Ephemeral run: `uvx --from git+https://github.com/jmchilton/planemo@a9b8b8bc7ab3b12035d53bdb5383fe450413d9f3 planemo`.
  Check: `planemo --version`.
  Docs: https://planemo.readthedocs.io/

## Run options

Optional flags, given as leading arguments. Strip any you recognize; treat the remaining positional argument as the run slug. Both default off and compose.

- `--use-subagents` — run each cast phase in its own subagent to keep this orchestrator's context small. For each phase whose skill is cast, spawn a subagent, tell it the run directory and to invoke the named skill with every default filename prefixed by `./<run-slug>/`, and have it return a short report (artifacts written, assumptions, status) rather than its full transcript; carry only that report forward. A cast loop phase runs **one subagent per iteration** — each advances a single step and returns its done-signal, and you inspect that signal to decide whether to spawn the next iteration. Branch phases run their whole fallback chain in one subagent. MANUAL (un-cast) phases are never delegated — including MANUAL loop phases — so handle those yourself regardless of the per-iteration rule above.
- `--checkpoint` — commit after every phase so the run directory's git history is a per-step record (a data source for workflow-implementation visualizations). When set, `git init ./<run-slug>/` during working-directory setup — this is a standalone per-run repo; do not add it to any surrounding repo you are working inside. Then after each phase's artifact is confirmed run `git -C ./<run-slug>/ add -A && git -C ./<run-slug>/ commit -m "phase <n>: <skill>"`. Loop phases commit **once per iteration** (`phase <n> step <k>: <skill>`); for a MANUAL loop, commit once per by-hand step. With `--use-subagents`, the subagent does the work and returns; you make the commit.

## Working directory (do this first)

Every constituent skill writes fixed filenames to its working directory. To keep one run's artifacts namespaced and avoid clobbering a prior run (foundry#282):

1. Pick a run slug — use the harness argument if given; else ask the user for a short project name up front (the directory must exist before phase 1 writes its first artifact, so don't wait for a source title); else default `nextflow-to-cwl-run`.
2. Create `./<run-slug>/` in the current directory. If it exists, suffix `-2`, `-3`, … . If invoked with `--checkpoint`, run `git init ./<run-slug>/` now (see Run options).
3. Run **every** skill invocation below with `./<run-slug>/` as its working directory: **prefix every default input and output filename with `./<run-slug>/`** when you invoke the skill. The skills preserve their declared basenames and honor a harness-supplied directory; you supply the prefix on **both reads and writes**, so each phase finds the prior phase's output and nothing lands in the repo root.

Announce the chosen directory before starting.

## Pipeline

Run these phases in order. After each, confirm the expected artifact exists in the run directory before advancing.

1. **summarize-nextflow** — invoke the `summarize-nextflow` skill. Read a Nextflow pipeline source tree (nf-core or ad-hoc DSL2) and emit a structured JSON summary for downstream translation Molds.
2. **nextflow-summary-to-cwl-interface** — MANUAL — `nextflow-summary-to-cwl-interface` is not yet cast. Map a Nextflow summary into a CWL Workflow interface design brief. Do this by hand and confirm before continuing.
3. **nextflow-summary-to-cwl-data-flow** — MANUAL — `nextflow-summary-to-cwl-data-flow` is not yet cast. Translate a Nextflow summary into a CWL data-flow design brief. Do this by hand and confirm before continuing.
4. **summary-to-cwl-template** — MANUAL — `summary-to-cwl-template` is not yet cast. CWL Workflow skeleton with per-step TODOs from source and design handoffs. Do this by hand and confirm before continuing.
5. **summarize-cwl-tool** (loop) — MANUAL — `summarize-cwl-tool` is not yet cast. Derive a CommandLineTool description (container, baseCommand, IO) for a CWL target. No shared endstate oracle yet; iterate over the tools enumerated in the CWL template, doing each by hand.
6. **implement-cwl-tool-step** (loop) — MANUAL — `implement-cwl-tool-step` is not yet cast. Convert an abstract step into a concrete CWL CommandLineTool + step. No shared endstate oracle yet; iterate over the tools enumerated in the CWL template, doing each by hand.
7. **validate-cwl** (loop) — MANUAL — `validate-cwl` is not yet cast. Run cwltool --validate / schema lint, classify failures, recommend fixes. No shared endstate oracle yet; iterate over the tools enumerated in the CWL template, doing each by hand.
8. **nextflow-test-to-cwl-test-plan** — MANUAL — `nextflow-test-to-cwl-test-plan` is not yet cast. Translate Nextflow test evidence into a CWL workflow test plan. Do this by hand and confirm before continuing.
9. **implement-cwl-workflow-test** — MANUAL — `implement-cwl-workflow-test` is not yet cast. Assemble CWL job file(s) and expected-output assertions. Do this by hand and confirm before continuing.
10. **validate-cwl** — MANUAL — `validate-cwl` is not yet cast. Run cwltool --validate / schema lint, classify failures, recommend fixes. Do this by hand and confirm before continuing.
11. **run-workflow-test** — invoke the `run-workflow-test` skill. Execute a workflow's tests via Planemo; emit structured pass/fail and outputs.
12. **debug-cwl-workflow-output** — MANUAL — `debug-cwl-workflow-output` is not yet cast. Triage failing CWL run outputs; classify failure modes; propose fixes. Do this by hand and confirm before continuing.

## Done

Report the final artifacts in `./<run-slug>/` and any phases handled manually (marked MANUAL above).

## Notes

- Do not re-implement any skill's internal logic here; this harness only sequences and routes.
- Carry unresolved assumptions forward as notes rather than inventing missing inputs.
- NF brings real test fixtures, so `nextflow-test-to-cwl-test-plan` replaces the `test-data-resolution` chain that paper-sourced pipelines need.