Gleam Multimodal Learner - HNSCC Recurrence Prediction with HANCOCK

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Gleam Multimodal Learner - HNSCC Recurrence Prediction with HANCOCK

<div class="contributors-line">
		
	
<ul class="text-list">
			
			<li>
				<a href="/training-material/hall-of-fame/paulocilasjr/" class="contributor-badge contributor-paulocilasjr"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/paulocilasjr?s=36" alt="Paulo Cilas Morais Lyra Junior avatar" width="36" class="avatar" />
    Paulo Cilas Morais Lyra Junior</a>
			<li>
				<a href="/training-material/hall-of-fame/qchiujunhao/" class="contributor-badge contributor-qchiujunhao"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/qchiujunhao?s=36" alt="Junhao Qiu avatar" width="36" class="avatar" />
    Junhao Qiu</a>
			<li>
				<a href="/training-material/hall-of-fame/khaivandangusf2210/" class="contributor-badge contributor-khaivandangusf2210"><img src="https://avatars.githubusercontent.com/khaivandangusf2210?s=36" alt="Khai Van Dang avatar" width="36" class="avatar" />
    Khai Van Dang</a>
			<li>
				<a href="/training-material/hall-of-fame/jgoecks/" class="contributor-badge contributor-jgoecks"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/jgoecks?s=36" alt="Jeremy Goecks avatar" width="36" class="avatar" />
    Jeremy Goecks</a></li>
</ul>

</div>

<div class="footnote" style="bottom: 8em;">
  <i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated:   
  <i class="fas fa-fingerprint" aria-hidden="true"></i><span class="visually-hidden">purl</span><abbr title="Persistent URL">PURL</abbr>: <a href="https://gxy.io/GTN:S00157">gxy.io/GTN:S00157</a>
</div>

<div class="footnote" style="bottom: 5em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> |

</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions

- How do we combine clinical, text, and image modalities to predict HNSCC recurrence?

- How do we configure Multimodal Learner to respect a predefined train/test split?

- How do we interpret ROC AUC and class-wise performance for recurrence prediction?

---

### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives

- Load HANCOCK metadata and CD3/CD8 image archives into Galaxy.

- Train a multimodal model with tabular, text, and image backbones.

- Evaluate test performance and compare to the HANCOCK benchmark.

---

# Introduction to GLEAM Multimodal Learner

- **Galaxy**: A web-based platform for data-intensive biomedical research
- **GLEAM Multimodal Learner**: No-code tool for joint modeling of tabular, text, and image data
- **Goal**: Predict head and neck cancer recurrence from the HANCOCK cohort

---

# Use Case: HANCOCK HNSCC Recurrence

- **Dataset**: HANCOCK multimodal cohort (763 patients)
- **Task**: Binary classification (recurrence vs no recurrence)
- **Modalities**: Clinical tabular variables, ICD text, CD3/CD8 TMA images

---

# Data Assets

- **Training table**: `HANCOCK_train_split.csv`
- **Test table**: `HANCOCK_test_split.csv`
- **Images archive**: `tma_cores_cd3_cd8_images.zip`
- **Main record**: https://zenodo.org/records/17933596

---

# Multimodal Modeling Strategy

| Modality | Source | Encoder |
|---|---|---|
| Tabular | Clinical + pathology + labs | FT-Transformer |
| Text | ICD codes (free text) | ELECTRA base |
| Image | CD3/CD8 TMA cores | CAFormer b36 |

- Late-fusion network combines modality embeddings
- Pretrained backbones reduce data requirements

---

# Tool Configuration

- **Training dataset**: filtered `dataset == training`
- **Test dataset**: filtered `dataset == test`
- **Text backbone**: `google/electra-base-discriminator`
- **Image backbone**: `caformer_b36.sail_in22k_ft_in1k`
- **Metric**: ROC AUC
- **CV**: 5-fold cross-validation
- **Threshold**: 0.25

---

# Outputs

- **HTML report**: metrics, ROC curves, confusion matrix
- **Metrics JSON**: per-split metrics and summary stats
- **Config YAML**: full run settings for reproducibility

---

# Results Summary

| Metric | HANCOCK (reference) | Multimodal Learner |
|---|---:|---:|
| ROC AUC | 0.79 | 0.74 |

- Performance is close to the published benchmark
- Class-wise metrics highlight stronger performance on the negative class

---

# Takeaways

- Multimodal Learner combines clinical, text, and imaging data in one run
- Predefined train/test split preserves benchmark comparability
- GLEAM provides reproducible configuration and transparent reports

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<div class="contributors-line">
		
<table class="contributions">
	
	<tr>
		<td><abbr title="These people wrote the bulk of the tutorial, they may have done the analysis, built the workflow, and wrote the text themselves.">Author(s)</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/paulocilasjr/" class="contributor-badge contributor-paulocilasjr"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/paulocilasjr?s=36" alt="Paulo Cilas Morais Lyra Junior avatar" width="36" class="avatar" />
    Paulo Cilas Morais Lyra Junior</a><a href="/training-material/hall-of-fame/qchiujunhao/" class="contributor-badge contributor-qchiujunhao"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/qchiujunhao?s=36" alt="Junhao Qiu avatar" width="36" class="avatar" />
    Junhao Qiu</a><a href="/training-material/hall-of-fame/khaivandangusf2210/" class="contributor-badge contributor-khaivandangusf2210"><img src="https://avatars.githubusercontent.com/khaivandangusf2210?s=36" alt="Khai Van Dang avatar" width="36" class="avatar" />
    Khai Van Dang</a><a href="/training-material/hall-of-fame/jgoecks/" class="contributor-badge contributor-jgoecks"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/jgoecks?s=36" alt="Jeremy Goecks avatar" width="36" class="avatar" />
    Jeremy Goecks</a>
		</td>
	</tr>

<tr class="reviewers">
		<td><abbr title="These people reviewed this material for accuracy and correctness">Reviewers</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/shiltemann/" class="contributor-badge contributor-badge-small contributor-shiltemann"><img src="https://avatars.githubusercontent.com/shiltemann?s=36" alt="Saskia Hiltemann avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/paulocilasjr/" class="contributor-badge contributor-badge-small contributor-paulocilasjr"><img src="https://avatars.githubusercontent.com/paulocilasjr?s=36" alt="Paulo Cilas Morais Lyra Junior avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/anuprulez/" class="contributor-badge contributor-badge-small contributor-anuprulez"><img src="https://avatars.githubusercontent.com/anuprulez?s=36" alt="Anup Kumar avatar" width="36" class="avatar" /></a></td>
	</tr>

</table>

</div>

</div>

Tutorial Content is licensed under <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br/>