Image annotations and model training are essential in bioimage analysis tasks. In biology and related fields, researchers often deal with large volumes of microscopy images that require accurate annotation to train machine learning models. Automate this process can save time and improve reproducibility, but high-quality training data remains critical. Human-in-the-loop workflows have emerged as a solution to bridge the gap between manual annotation and automated model training, enabling iterative improvements through user interaction.

This tutorial introduces a reusable human-in-the-loop image segmentation and training workflow implemented in Galaxy, designed to streamline the creation of custom segmentation models using YOLO (You Only Look Once). The workflow integrates both interactive and headless tools. We demonstrate the workflow here with tail analysis images from growing embryo samples. The workflow can also be adapted to other datasets with minimal adjustments.

Agenda

In this tutorial, we will cover:

1. Get data
2. Preprocessing the images
   1. Perform histogram equalization
   2. Convert image format
3. Annotating the images
4. Prepare training data
5. Train a YOLO model
6. Extract complete training workflow from history
7. Using the trained model to label new images
8. Conclusion

Get data

Hands On: Data Upload

1. Create a new history for this tutorial

2. Import the following dataset from Zenodo or from the data library.
   • Important: Choose the type of data as zip.

   https://zenodo.org/records/16096782/files/example-input-images.zip
   

   Tip: Importing via links

   • Copy the link location

   • Click galaxy-upload Upload Data at the top of the tool panel

   • Select galaxy-wf-edit Paste/Fetch Data

   • Paste the link(s) into the text field

   • Press Start

   • Close the window

   Tip: Importing data from a data library

   As an alternative to uploading the data from a URL or your computer, the files may also have been made available from a shared data library:

   1. Go into Libraries (left panel)
   2. Navigate to the correct folder as indicated by your instructor.
      • On most Galaxies tutorial data will be provided in a folder named GTN - Material –> Topic Name -> Tutorial Name.
   3. Select the desired files
   4. Click on Add to History galaxy-dropdown near the top and select as Datasets from the dropdown menu

   5. In the pop-up window, choose

      • “Select history”: the history you want to import the data to (or create a new one)
   6. Click on Import

3. Unzip ( Galaxy version 6.0+galaxy0) with the following parameters:
   • param-file “input_file”: example-input-images.zip
   • “Extract single file”: All files
   • Click on Edit galaxy-pencil next to the collection name in your history to rename it to input-images

4. Import the class name file from Zenodo

   https://zenodo.org/records/16096782/files/class_names.txt
   

   Tip: Importing via links

   • Copy the link location

   • Click galaxy-upload Upload Data at the top of the tool panel

   • Select galaxy-wf-edit Paste/Fetch Data

   • Paste the link(s) into the text field

   • Press Start

   • Close the window

5. Inspect the content of class_names.txt.
   • Click on the title of your file to see the row of small icons for saving, linking, etc.:
   • Click on the visualise icon galaxy-visualise and then select the Editor visualization plugin.

Question

1. How many classes are there in this class file?
2. What are the class names?

Solution

1. 1
2. tail

Preprocessing the images

The example dataset used in this tutorial consists of images in TIFF format. However, the YOLO training tool in Galaxy currently supports only JPEG (JPG) images. Therefore, as a necessary preprocessing step, the input images must be converted to the appropriate format. In this workflow, we first apply histogram equalization to the images and then convert them from TIFF to JPG.

It’s important to note that these preprocessing steps are tailored to the example data. When applying this workflow to other images, the specific preprocessing steps may vary. Format conversion is mandatory if the images are not already in JPG, but contrast enhancement or other adjustments may or may not be needed depending on images. Users should adapt this part of the workflow to best suit their own data.

Perform histogram equalization

Hands On: Normalize Histogram

1. Perform histogram equalization ( Galaxy version 0.18.1+galaxy0) with the following parameters:
   • param-collection “Input image”: input-images (Input dataset collection)
   • “Histogram equalization algorithm”: CLAHE
2. Rename galaxy-pencil the output collection to input-normalized.

Convert image format

Hands On: Convert Format

1. Convert image format ( Galaxy version 1.3.45+galaxy0) with the following parameters:
   • param-file “Image to convert”: input-normalized (Input dataset collection, output of Perform histogram equalization tool)
   • “Transformations”: Deselect all
   • “Reduce Color Palette”: No
   • “Resize(%)”: 100.0
   • “Output Format”: jpg
2. Rename galaxy-pencil the output collection to input-converted.

Annotating the images

Once the input images are preprocessed and converted to JPG format, we will need to manually annotate the images using AnyLabeling. AnyLabeling is an interactive labeling tool integrated into Galaxy that allows users to draw bounding boxes or other shapes around objects of interest.

In this tutorial, we annotate all 22 example images using the tool’s Auto labeling function, which helps accelerate the annotation process. The labels will later be used to train the YOLO model. The annotation process is entirely human-in-the-loop, so the quality and relevance of the labels directly influence the performance of the resulting model.

Hands On: Interactive Annotation

1. Start AnyLabeling Interactive with the following parameters:
   • param-file “Input images”: input-converted (Input dataset collection, output of Convert image format tool)
   • “Pre-existing annotations”: Empty
   • “Classes file”: Empty
   • “A tarball containing custom model files and yaml files”: Empty

   Tip: Open interactive tool

   1. Go to User > Active InteractiveTools
   2. Wait for the to be running (Job Info)
   3. Click on

2. Once AnyLabeling is open.

   • Navigate to File -> Change Output Dir and set the output directory to working/home/output.

   • To open your input images, go to File -> Open Dir, and choose working/home/input_images directory.

• Click the Auto Labeling button, choose Segment Anything 2 (Hiera-Tiny) from the model dropdown, select +Point,then begin annotation by clicking on the object in the image. Wait for the inference to complete before continuing.

• If the inferred polygon is not accurate, click -Point, then click on a background (non-object) area to refine the results. The model will re-run the inference.

• After the inference is complete, click Finish Object. In the Enter object label box, type tail then click OK to save the annotation.

• Important: The object label must match the content of class_names.txt.
• Repeat these steps for each image in the dataset to complete the interactive annotation process.

Prepare training data

In this step, we will convert the annotation file generated by AnyLabeling (in JSON format) into YOLO compatible TXT files. This conversion is necessary because the YOLO training tool expects annotations in its specific text format, where each object is described by a class ID and corresponding bounding box coordinates.

Hands On

1. Convert AnyLabeling JSON to YOLO text ( Galaxy version 8.3.0+galaxy2) with the following parameters:
   • param-file “Input label files”: al_output (output of AnyLabeling Interactive tool)
   • param-file “Class file”: class_names.txt (Input dataset)
2. Rename galaxy-pencilthe output collection to yolo-files

Train a YOLO model

Hands On:

In this step, we will train a custom YOLO model using the prepared images and annotation files.

1. Perform YOLO training ( Galaxy version 8.3.0+galaxy2) with the following parameters:
   • param-file “Input images”: input-converted (output of Convert image format tool)
   • param-file “Input YOLO txt files”: yolo-files (output of Convert AnyLabeling JSON to YOLO text tool)
   • param-file “Model URL”: YOLO11n-seg
   • In “Training Parameters”:
     • “How do you want to split your images for training.”: 70
     • “Number of epochs for taining.”: 50
     • “Image size”: 512
     • “Image scale augmentation”: 0.8
     • “Image rotation augmentation”: 10.0
     • “Image HSV-Value”: 0.5
     • “Image HSV-Saturation”: 0.7
     • “Image HSV-Hue”: 0.015
     • “Learning rate”: 0.02
     • “Weight decay”: 0.001
     • “Confidence”: 0.5
     • “IoU”: 0.7
     • “Max. number of detection”: 300

Question

1. What output files are generated after training?
2. How do training parameters affect model performance?
3. What happen if the Confidence threshold increase or decrease?

Solution

1. The training step produces the best model, last model, training metrics, and a training plot.

2. Number of epochs: A higher number allows the model to learn more thoroughly but takes longer.

   Image size: Determines the resolution used during training. Larger sizes can improve accuracy but require more memory and time.

   Image scale augmentation: Applies random resizing of images during training to improve robustness. Values between 0.5 and 1.0 are commonly used.

   Image rotation augmentation: Helps the model generalize to rotated objects. Keep small (e.g., 5–15°) for biological images to avoid distortion.

   Image HSV-Value: Randomly adjusts image brightness and color during training. Can help with generalization, especially when image lighting varies.

   Learning rate: Controls how fast the model learns. If training is unstable or not improving, try lowering this (e.g., to 0.01 or 0.005).

   Weight decay: Helps prevent overfitting by penalizing large weights. Usually works well between 0.0001 and 0.01.

3. Increasing the confidence threshold results in fewer, more reliable segmentations, while decreasing it may produce more segments, including uncertain or noisy ones.

Extract complete training workflow from history

As an optional step, you can extract a complete workflow from your Galaxy history. This allows you to save and reuse the entire training process as a reproducible and shareable workflow.

Tip: Extracting a workflow from your history

1. Clean up your history: remove any failed (red) jobs from your history by clicking on the galaxy-delete button.

   This will make the creation of the workflow easier.

2. Click on galaxy-gear (History options) at the top of your history panel and select Extract workflow.

   

   The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.

3. Replace the Workflow name to something more descriptive.

4. Rename each workflow input in the boxes at the top of the second column.

5. If there are any steps that shouldn’t be included in the workflow, you can uncheck them in the first column of boxes.

6. Click on the Create Workflow button near the top.

   You will get a message that the workflow was created.

Using the trained model to label new images

Once training is complete, the resulting model can be used to label new images. The process of applying the trained model for inference is covered in a separate tutorial. Please refer to that tutorial for detailed instructions on how to use the model for prediction.

Conclusion

This tutorial demonstrated how to train a custom YOLO model for object detection and segmentation in bioimages using Galaxy. By combining image preprocessing, human-in-the-loop annotation, format conversion, and model training, we have built a complete workflow. We can apply the trained model to new image data and further refine our workflow as needed.

You've Finished the Tutorial

Please also consider filling out the Feedback Form as well!

Key points

• Image preprocessing (e.g., histogram equalization) can enhance model performance but may not be necessary for all datasets

• The accuracy and consistency of human annotations directly impact the quality of the trained YOLO model.

• Training metrics, the output of the YOLO training tool, are essential for evaluating model performance and guiding improvements.

Frequently Asked Questions

• Tutorial FAQs
• Topic FAQs
• Galaxy FAQs
• GTN Matrix Chat
• Galaxy Help Forum

Useful literature

Further information, including links to documentation and original publications, regarding the tools, analysis techniques and the interpretation of results described in this tutorial can be found here.

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.
Did you use this material as a learner or student? Click the form below to leave feedback.

Citing this Tutorial

1. Yi Sun, Arif UI Maula Khan, Training Custom YOLO Models for Object Detection and Segmentation in Bioimages (Galaxy Training Materials). https://training.galaxyproject.org/training-material/topics/imaging/tutorials/yolo-segmentation-training/tutorial.html Online; accessed TODAY
2. Hiltemann, Saskia, Rasche, Helena et al., 2023 Galaxy Training: A Powerful Framework for Teaching! PLOS Computational Biology 10.1371/journal.pcbi.1010752
3. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012

BibTeX

@misc{imaging-yolo-segmentation-training,
author = "Yi Sun and Arif UI Maula Khan",
	title = "Training Custom YOLO Models for Object Detection and Segmentation in Bioimages (Galaxy Training Materials)",
	year = "",
	month = "",
	day = "",
	url = "\url{https://training.galaxyproject.org/training-material/topics/imaging/tutorials/yolo-segmentation-training/tutorial.html}",
	note = "[Online; accessed TODAY]"
}
@article{Hiltemann_2023,
	doi = {10.1371/journal.pcbi.1010752},
	url = {https://doi.org/10.1371%2Fjournal.pcbi.1010752},
	year = 2023,
	month = {jan},
	publisher = {Public Library of Science ({PLoS})},
	volume = {19},
	number = {1},
	pages = {e1010752},
	author = {Saskia Hiltemann and Helena Rasche and Simon Gladman and Hans-Rudolf Hotz and Delphine Larivi{\`{e}}re and Daniel Blankenberg and Pratik D. Jagtap and Thomas Wollmann and Anthony Bretaudeau and Nadia Gou{\'{e}} and Timothy J. Griffin and Coline Royaux and Yvan Le Bras and Subina Mehta and Anna Syme and Frederik Coppens and Bert Droesbeke and Nicola Soranzo and Wendi Bacon and Fotis Psomopoulos and Crist{\'{o}}bal Gallardo-Alba and John Davis and Melanie Christine Föll and Matthias Fahrner and Maria A. Doyle and Beatriz Serrano-Solano and Anne Claire Fouilloux and Peter van Heusden and Wolfgang Maier and Dave Clements and Florian Heyl and Björn Grüning and B{\'{e}}r{\'{e}}nice Batut and},
	editor = {Francis Ouellette},
	title = {Galaxy Training: A powerful framework for teaching!},
	journal = {PLoS Comput Biol}
}

                   

Galaxy Administrators: Install the missing tools

You can use Ephemeris's shed-tools install command to install the tools used in this tutorial.

shed-tools install [-g GALAXY] [-a API_KEY] -t <(curl https://training.galaxyproject.org/training-material/api/topics/imaging/tutorials/yolo-segmentation-training/tutorial.json | jq .admin_install_yaml -r)

Alternatively you can copy and paste the following YAML

---
install_tool_dependencies: true
install_repository_dependencies: true
install_resolver_dependencies: true
tools:
- name: graphicsmagick_image_convert
  owner: bgruening
  revisions: 8f46605c84ec
  tool_panel_section_label: Convert Formats
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: 2d_histogram_equalization
  owner: imgteam
  revisions: b1c2c210813c
  tool_panel_section_label: Imaging
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: unzip
  owner: imgteam
  revisions: 57f0914ddb7b
  tool_panel_section_label: Collection Operations
  tool_shed_url: https://toolshed.g2.bx.psu.edu/

Feedback

No feedback has been recieved yet for this training. Be the first one by filling in the feedback form.