Galaxy Apply Rules DSL

Galaxy’s __APPLY_RULES__ reorganizes a collection by deriving columns from element identifiers and tags, transforming those columns, and mapping them into a new collection. It can relabel, filter, sort, flatten, or change nesting without processing file contents. The output is named output.

Use a dedicated collection tool when it expresses the operation directly, such as flattening or extracting an element. Apply Rules fits transformations that depend on parsed identifiers, tag values, or several operations together. galaxy-collection-tools lists the alternatives.

A rule set inside a workflow

The tool parameter named rules holds an object with two lists: rules performs the transformations and mapping defines the output. Mapping entries contain a type and a columns array. A sibling tool_state.mapping or a bare tool_state.rules array is not this serialized shape.

This step swaps the axes of a list:list input:

tool_id: __APPLY_RULES__
in:
  input: sample_by_segment
tool_state:
  input: { __class__: ConnectedValue }
  rules:
    rules:
      - type: add_column_metadata
        value: identifier0
      - type: add_column_metadata
        value: identifier1
    mapping:
      - type: list_identifiers
        columns: [1, 0]

The first rule creates column 0 from the outer identifier. The second creates column 1 from the inner identifier. Mapping [1, 0] groups by segment, then sample. The executable workflow and test live under verification/workflows/collection-swap-nesting-with-apply-rules/. See collection-swap-nesting-with-apply-rules for the expected collection.

Rows, columns, and sources

For collection input, Galaxy starts with one empty row per leaf dataset. Source metadata accompanies each row: its identifier path, index path, dataset reference, dataset tags, and any inherited sample-sheet columns. identifier0 means the outermost identifier. index0 is its zero-based position.

Column indices are zero-based and refer to the table at that point in execution. Addition rules append columns. Removing columns shifts subsequent indices. Filters and sorting carry source metadata along with the rows, so a renamed row still points to its original dataset.

Mapping uses the final table. Unmapped columns can remain, so removing intermediate columns is optional. Plan the output identifier path before writing rules, then track the columns that supply it.

Rule operations

Each rule requires type and the parameters listed below, except fields explicitly marked optional. Rules run in order. Names and behavior here follow Galaxy’s implementation and rule specification.

Add columns

Regex extraction uses Python regular expressions and searches within the cell. Without replacement or group_count, it appends the whole match, not the first capture group. With group_count, it appends one column per capture group and requires exactly that many groups. With replacement, it appends the expanded template for the match, such as '\1', not a rewritten copy of the whole cell.

Unmatched rows fail by default. allow_unmatched: true appends empty strings instead. Anchor expressions when the entire identifier must conform, and use YAML single quotes to preserve regex backslashes. collection-split-identifier-via-rules shows a nesting example.

Filter rows

invert is a required boolean for the first four filters. Its meaning depends on the operation:

Exact matching is case-sensitive and includes whitespace. Regex filters also search within cells.

add_filter_compare requires integer target_column, integer value, and compare_type: less_than, less_than_equal, greater_than, or greater_than_equal. It parses cells as floating-point numbers and keeps rows satisfying the comparison. It has no inversion parameter.

Reorder or reshape the table

For example, splitting groups [0] and [1] transforms ["left", "right", "sample"] into ["left", "sample"] and ["right", "sample"]. It does not generate a Cartesian product.

Map columns into a collection

A second list column creates another list level even if its values are forward and reverse. To create list:paired, use both list_identifiers and paired_identifier. Pair roles are case-insensitive: f, 1, r1, and forward normalize to forward, while r, 2, r2, and reverse normalize to reverse. For mixed paired and single reads, supply u or unpaired explicitly for single reads.

For flat identifiers sampleA_R1.fastq.gz and sampleA_R2.fastq.gz, this parameter derives the sample and pair roles directly:

rules:
  rules:
    - type: add_column_metadata
      value: identifier0
    - type: add_column_regex
      target_column: 0
      expression: '^(.+)_R([12])\.fastq\.gz$'
      group_count: 2
  mapping:
    - type: list_identifiers
      columns: [1]
    - type: paired_identifier
      columns: [2]

Column 1 is sampleA, column 2 is 1 or 2, and the output type is list:paired. Each sample must have the intended forward and reverse datasets. collection-build-list-paired-with-apply-rules describes paired promotion from existing nesting.

Check the result

Build a complex rule set incrementally. After each addition, inspect the table and retained identifiers before adding the next rule. Keep a column map in comments so later edits can update every affected index.

Test identifiers, collection type, and dataset membership together. A correct element count can hide swapped samples or incorrect pairing. Check representative contents as well as structure.

Watch for these failures:

• Duplicate output paths: two rows mapping to the same full identifier path cause the later dataset to replace the earlier one in the builder. Make the final path unique for every intended dataset.
• Unexpected filtering: inspect retained identifiers, especially around invert and count filters. Filtering is selection, not validation that every input conforms.
• All rows removed: check that filters retain the expected datasets, including cases with no matches. If no row survives, there is no dataset left to map. allow_unmatched only affects regex extraction and cannot prevent a filter from removing rows.
• Regex escaping: . matches any character. To match a literal filename extension, write a single-quoted YAML expression such as ‘.fastq$’. To match literal parentheses, use ‘(sample)’. Avoid double-quoted YAML unless you also account for its backslash escaping.
• Case differences: exact-match filters distinguish sample1 from Sample1. If case should be ignored, use a regex such as ’(?i)^sample1$’ with add_filter_regex.
• Missing metadata or bad indices: check the input nesting and recalculate column numbers after removal or splitting. Remove temporary columns near the end when that makes index tracking easier.
• Failed extraction or numeric conversion: fix the identifier contract or select the intended rows before extraction. Use allow_unmatched only when an empty result has a defined role.

The collection builder defines grouping and tag assignment. galaxy-collection-semantics covers downstream mapping and reduction. For a relabeling task, see relabel-via-rules-and-find-replace.