Galaxy 101

Overview

question Questions
  • Which coding exon has the highest number of single nucleotide polymorphisms (SNPs) on human chromosome 22?

objectives Objectives
  • Familiarize yourself with the basics of Galaxy

  • Learn how to obtain data from external sources

  • Learn how to run tools

  • Learn how histories work

  • Learn how to create a workflow

  • Learn how to share your work

time Time estimation: 1 hour 30 minutes

level Level: Introductory level level level

Supporting Materials

Introduction

This practical aims to familiarize you with the Galaxy user interface. It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows, and sharing your work.

comment Note: results may vary

Your results may be slightly different from the ones presented in this tutorial due to differing versions of tools, reference data, external databases, or because of stochastic processes in the algorithms.

Agenda

In this tutorial, we will:

  1. Pretreatments
    1. Locating Exons
    2. Obtaining SNP information
  2. Analysis
    1. Find exons with the most SNPs
    2. Count the number of SNPs per exon
    3. Sort the exons by SNPs count
    4. Select the top five exons
    5. Recovering exon info
    6. Displaying data in UCSC genome browser
  3. Galaxy management
    1. Convert your analysis history into a workflow
    2. The workflow editor
    3. Run workflow on different data
    4. Share your work

Pretreatments

We start with the question: In human chromosome 22, which exon has the highest number of single nucleotide polymorphisms (SNPs)?

comment Background

Not everyone has the same background and that’s ok! You may have studied different organisms with different gene models, or you may not be familiar with the biological aspect at all. The biological background is not necessary for following this tutorial, we just like to provide research context for any tutorial. Here are brief explanations of some of the concepts mentioned in this tutorial:

  • Nucleotides are the A, C, T, and Gs that make up DNA.
  • A Chromosomes can be thought of as a very long piece of DNA (string of A, C, T, Gs) Some organisms have chromosomes, in this tutorial we will use Human chromosome number 22.
  • Features are regions of the chromosome that are interesting for one reason or another. Some examples of features include genes, terminators, transcription start sites, and repeat regions.
  • Genes are one kind of interesting feature, a region that will be transcribed into RNA before being translated into proteins.
  • Exons are part of the eukaryotic gene model. A gene is a complex structure with numerous segments that are removed before the the gene is transcribed into an RNA. The exons are the coding regions, the regions that will appear in the final protein product.

    Eukaryotic gene model
    Figure 1: An original piece of DNA containing introns and exons has the introns cut out before the exons are joined together to form the mRNA. (Image from WikiMedia, under the Public Domain)
  • SNP stands for single-nucleotide polymorphism and is pronounced like the English word “snip”. SNPs are single nucleotide differences between a sequenced individual compared to some reference sequence; where one individual might have an A, another could have a C in that position. Databases of SNPs have been created for many organisms and they include any single nucleotide deviation from the reference sequence which appears in some percentage of the population (e.g. >1%). These are often especially interesting to geneticists as the causes of certain inheritable diseases.

You may be familiar with the UCSC Genome Browser or another resource like it, and know that you can find the data there. But even with your data in hand, you stll have the question: “how do I actually compute this?” There is really a straightforward answer: Galaxy. So let’s try it…

Browse to your favourite Galaxy instance and log in or register. The Galaxy interface consists of three main parts:

  1. The available tools are listed on the left
  2. Your analysis history is recorded on the right
  3. The central panel will let you run analyses and view outputs

Galaxy interface

hands_on Hands-on: Create history

  1. Make sure you start from an empty analysis history.

    tip Tip: Creating a new history

    Click the new-history icon at the top of the history panel

    If the new-history is missing:

    1. Click on the galaxy-gear icon (History options) on the top of the history panel
    2. Select the option Create New from the menu
  2. Rename your history to be meaningful and easy to find. You can do this by clicking on the title of the history (which by default is Unnamed history) and typing Galaxy 101 as the name. Do not forget to hit the Enter key on your keyboard to save it.

    Rename the history

Locating Exons

First we need to get some data into our history. You can upload files from your computer, or Galaxy can also fetch data directly from external sources. We know UCSC has exon locations for humans and we can use Galaxy to import the data for chromosome 22, directly from the UCSC table browser.

hands_on Hands-on: Data upload from UCSC

  1. UCSC Main tool - table browser:

    In the tool menu, navigate to Get Data -> UCSC Main - table browser

    `UCSC Main table browser` menu entry

    You will be taken to the UCSC table browser, which looks something like this:

    `UCSC table browser` tool, first screen for exons

    Now set the following parameters:

    • “clade”: Mammal
    • “genome”: Human
    • “assembly”: Dec. 2013 (GRCh38/hg38)
    • “group”: Genes and Gene Predictions
    • “track”: GENCODE v29
    • “table”: knownGene
    • param-text “region” should be changed to position with value chr22
    • “output format” should be changed to BED - browser extensible data
    • param-check “Send output to” should have the option Galaxy checked
  2. Click on the get output button and you will see the next screen:

    `UCSC table browser` tool, second screen for exons

    Change Create one BED record per to Coding Exons and then click on the Send query to Galaxy button.

    comment Comment

    After this you will see your first history item in Galaxy’s right panel. It will go through the gray (preparing/queued) and yellow (running) states to become green (success):

    `UCSC Main on Human: knownGene` dataset is green

  3. When the dataset is green, click on the galaxy-eye (eye) icon to view the contents of the file. It should look something like this:

    Contents of the `UCSC Main on Human: knownGene` dataset

    Each line represents an exon, the first three columns are the genomic location, and the fourth column contains the name of the exon.

  4. Let’s rename our dataset to something more recognizable.

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, change the Name field to Exons
    • Click the Save button

    Your history should now look something like this:

    Rename dataset to `Exons`

Obtaining SNP information

We now have information about the exon locations, but our original question was which exon contains the largest number of SNPs, so let’s get some information about SNP locations from UCSC as well:

hands_on Hands-on: SNP information

Again open the UCSC Main - table browser tool and set the following parameters

  1. UCSC Main tool table browser:
    • “group” should be changed to Variation
    • param-text “region” should be changed again to position with value chr22
    • “output format” should be changed again to BED - browser extensible data

    `UCSC table browser` tool, first screen for SNPs

    The “track” setting shows the version of the SNP database to get. In this example it is version 150, but you may select the latest one. Your results may vary slightly from the ones in this tutorial when you select a different version, but in general it is a good idea to select the latest version, as this will contain the most up-to-date SNP information.

  2. Click on the get output button to find a form similar to this:

    `UCSC table browser` tool, second screen for SNPs

    Make sure that “Create one BED record per” is set to Whole Gene (Whole Gene here really means Whole Feature), and click on Send query to Galaxy. A second item will appear in your analysis history.

  3. Now rename your new dataset to SNPs so we can easily remember what the file contains.

    tip Tip: Renaming a dataset

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, change the Name field to SNPs
    • Click the Save button

Analysis

Find exons with the most SNPs

Our objective is to find which exon contains the most SNPs. Therefore we have to intersect the file with the exon locations with the file containing the SNP locations (here “intersect” is just a fancy word for printing SNPs and exons that overlap side-by-side).

Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the search bar at the top of the tool panel to find your tool.

Additionally different servers may have multiple, similarly named tools which accomplish similar functions. For these tutorials, you should select precisely the one that is described. However, in your real analyses, you’ll need to search among the various options to find the one that works for you.

hands_on Hands-on: Finding Exons

  1. Join tool the intervals of two datasets side-by-side:

    Enter the word join in the search bar of the tool panel, and select the tool named Join - the intervals of two datasets side-by-side

    • “Join”: Select Exons
    • “With”: Select SNPs
    • “with min overlap”: 1
    • “Return”: Only records that are joined (INNER JOIN), which means that only matches are included in the output (i.e. only exons with SNPs in it and only SNPs that fall in exons)

    tip How do I use this tool?

    All Galaxy tools include documentation. If you scroll down on this page, you will find the help of the tool.

  2. Click Execute
  3. Wait for the job to finish
  4. View the resulting file (with the galaxy-eye (eye) icon). If everything went okay, you should see a file that looks similar to this:

    Contents of the `Join` output dataset

    Remember that variations are possible due to using different versions of UCSC databases: as long as you have similar looking columns you did everything right!

comment If things didn’t work…

Did the Join tool error with a memory failure? Or is this step executing for a long time? Most likely a setting was missed when extracting the data from the UCSC Table Browser. Try again, double checking that:

  • For both SNP and EXON: “region” is actually changed to position with value chr22
  • For EXON: “Create one BED record per” Coding Exons is selected (not Whole Gene as for the SNP data)
  • Carefully inspect the remaining Table Browser settings if these two most common reasons for problems were correct in your tool executions

Let’s take a look at this dataset. The first six columns correspond to the exons, and the last six columns correspond to the SNPs. Column 4 contains the exon IDs, and column 10 contains the SNP IDs. In our screenshot you see that the first lines in the file all have the same exon ID but different SNP IDs, meaning these lines represent different SNPs that all overlap the same exon.

question Question

For the first 3 exons in your file, what is the number of SNPs that fall into that exon?

solution Solution

At the time of writing, for hg38/GENCODE v29, joined with “Common SNPs(151)”, using ctrl-f to look for how many times each is used:

Gene Occurences
ENST00000252835.5_cds_0_0_chr22_15528159_f 24
ENST00000643195.1_cds_0_0_chr22_15528192_f 24
ENST00000343518.10_cds_0_0_chr22_15690078_f 40

Count the number of SNPs per exon

Since each line in our file represents a single overlap between SNP and exon, we can find the total number of SNPs in an exon, simply by counting the number of lines that have the same exon ID. So let’s do this for all the exons in our file:

hands_on Hands-on: Counting SNPs

  1. Group tool data by a column and perform aggregate operation on other columns:

    • “Select data”: select the output dataset from Join tool
    • “Group by column”: Column: 4 (the column with the exon IDs)
    • “Insert Operation”:
      • 1: Operation
        • “Type”: Count
        • “On column”: Column: 4

    Settings for the `Group` tool

  2. Click Execute to perform the grouping. Your new output dataset will look something like this:

    Contents of the `Group` output dataset

This file contains only two columns. The first contains the exon IDs, and the second the number of times that exon ID appeared in the file - in other words, how many SNPs were present in that exon.

question Question

How many exons are there in total in your file?

solution Solution

Each line now represents a different exon, so you can see the answer to this when you expand the history item, as in the image above. The exact number you see for your dataset may be slightly different due to the updates to the exon and SNPs information in UCSC.

Sort the exons by SNPs count

Now that we have a list of all exons, and the number of SNPs they contain, we would like to know which exon has the highest number of SNPs. We can do this by sorting the file on the second column.

hands_on Hands-on: Sorting

  1. Sort tool data in ascending or descending order:

    • “Sort Query”: Output from Group tool
    • “Column Selections”:
      • 1: Column Selections
        • “on column”: Column: 2
        • “in”: Descending order
        • “Flavor”: Fast numeric sort (-n)
  2. Click Execute

  3. Examine the output file.

    Contents of the `Sort` output dataset

    The file should look similar to before sorting, but now the exons with the highest number of SNPs are at the top.

question Question

Which exon has the highest number of SNPs in your file?

solution Solution

When this tutorial was last updated, ENST00000343518.10_cds_0_0_chr22_15690078_f had 40 SNPs. Keep in mind this may depend on your settings when getting the data from UCSC.

Select the top five exons

Let’s say we want a list with just the top-5 exons with highest number of SNPs.

hands_on Hands-on: Select first

  1. Select first tool lines from a dataset:

    • “Select first”: 5
    • “from”: The output from Sort tool
  2. Click Execute

  3. Examine the output file, this should contain only the first 5 lines of the previous dataset.

    Contents of the `Select first` output dataset

Recovering exon info

Congratulations! You have now determined which exons on chromosome 22 have the highest number of SNPs, but what else can we learn about them? One way to learn more about a genetic location is to view it in a genome browser. However, in the process of getting our answer, we have lost information about the location of these exons on the chromosome. But fear not, Galaxy saves all of your data, so we can recover this information quite easily.

hands_on Hands-on: Compare two Datasets

  1. Compare two Datasets tool to find common or distinct rows:

    • “Compare”: Exons
    • “Using column”: Column: 4
    • “against”: the output from Select first tool
    • “Using column”: Column: 1
    • “to find”: Matching rows of 1st dataset
  2. Click Execute
  3. Examine your output file. It should contain the locations of your top 5 exons:

    Contents of the `Compare two Datasets` output dataset

Displaying data in UCSC genome browser

A good way to learn about these exons is to look at their genomic surrounding. This can be done by using genome browsers. Galaxy can launch a genome browser such as IGV on your local machine, and it can connect to online genome browsers as well. An example of such an online genome browser is the UCSC genome browser.

hands_on Hands-on: UCSC genome browser

  1. First, check that the database of your latest history dataset is hg38. If not, click on the galaxy-pencil pencil icon and modify the Database/Build: field to Human Dec. 2013 (GRCh38/hg38) (hg38).

    tip Tip: Changing Database/Build (dbkey)

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, change the Database/Build field
    • Select your desired database key from the dropdown list: hg38
    • Click the Save button
  2. To visualize the data in UCSC genome browser, click on display at UCSC main option visible when you expand the history item.

    `display at UCSC main` link

    This will upload the data to UCSC as custom track. To see your data look at the User Track near the top. You can enter the coordinates of one of your exons at the top to jump to that location.

    `User Track` shown in the UCSC genome browser

UCSC provides a large number of tracks that can help you get a sense of your genomic area, it contains common SNPs, repeats, genes, and much more (scroll down to find all possible tracks).

Galaxy management

In Galaxy your analyses live in histories such as your current one. Histories can be very large, and you can have as many histories as you want. You can control your histories (switching, copying, sharing, creating a fresh history, etc.) in the galaxy-gear History Options menu on the top of the history panel:

History options menu

If you create a new history, your current history does not disappear. If you would like to list all of your histories just use the galaxy-columns Multi-history view, or choose Saved Histories from the history options menu and you will see a list of all your histories in the central panel:

`Saved Histories` list

An alternative overview of your histories can be accessed by clicking on the galaxy-columns View all histories button at top of your history panel.

Overview of histories with their datasets

Here you see a more detailed view of each history, and can perform the same operations as the other view:

  • switching between different histories,
  • deleting a history,
  • purging it (permanently deleting it, this action cannot be reversed)
  • copying histories

One difference is that this view lets you copy datasets between your histories by dragging and dropping them.

You can always return to your analysis view by clicking on Analyze Data in the top menu bar.

Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step. But when you receive new data, or a new report is requested, it would be tedious to do each step over again. Wouldn’t it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the Extract workflow option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

hands_on Hands-on: Extract workflow

  1. Clean up your history. If you had any failed jobs (red), please remove those datasets from your history by clicking on the x button. This will make the creation of a workflow easier.

  2. Go to the history galaxy-gear History Options menu and select the Extract Workflow option.

    `Extract Workflow` entry in the history options menu

    The central panel will change as shown below and you will be able to choose which steps to include/exclude and how to name the newly created workflow.

    Selection of steps for `Extract Workflow` from history

  3. Rename the workflow to something descriptive, for example Find exons with the highest number of interactions.

    While we created this workflow initially to analyse SNPs, if we had similarly formatted datasets we could use this workflow to find those interactions as well.

    If there are any steps that shouldn’t be included in the workflow, you can uncheck them.

  4. Click on the Create Workflow button near the top.

    You will get a message that the workflow was created. But where did it go?

  5. Click on Workflow in the top menu of Galaxy. Here you have a list of all your workflows. Your newly created workflow should be listed at the top:

    `Your workflows` list

The workflow editor

We can examine the workflow in Galaxy’s workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.

hands_on Hands-on: Extract workflow

  1. Click on the triangle to the right of your workflow name.

    `Edit` option in the menu for the latest workflow

  2. Select Edit to launch the workflow editor. You should see something like this:

    Workflow editor

    When you click on a workflow step, you will get a view of all the parameter settings for that tool on the right-hand side of your screen.

    tip Hiding intermediate steps

    When a workflow is executed, the user is usually primarily interested in the final product and not in all intermediate steps. By default all the outputs of a workflow will be shown, but we can explicitly tell Galaxy which outputs to show and which to hide for a given workflow. This behaviour is controlled by the little asterisk next to every output dataset: Asterisk for `out_file1` in the `Select First` tool

    If you click on this asterisk for any of the output datasets, then only files with an asterisk will be shown, and all outputs without an asterisk will be hidden. (Note that clicking all outputs has the same effect as clicking none of the outputs, in both cases all the datasets will be shown.)

  3. Click the asterisk for out_file1 in the Select First and Compare two Datasets tools.

    Now, when we run the workflow, we will only see the final two outputs, i.e. the table with the top-5 exons and their SNP counts, and the file with exons ready for viewing in a genome browser. Once you have done this, you will notice that the minimap at the bottom-right corner of your screen will have a colour-coded view of your workflow, with orange boxes representing a tool with an output that will be shown.

    Workflow minimap

    In the image above, you see that the top input dataset (with the blue border) connects to the first input of the Join tool, so this corresponds to the exon data.

  4. The box named Exons is named ok, but we want to change SNPs since this workflow is not specific to SNPs

    • Click on the box corresponding to the SNPs input dataset
    • change the Label to Features on the right-hand side of your screen.
  5. Let’s also rename the outputs:

    • Click on the Select first tool in the workflow editor
    • In the menu on the right click on Configure Output: 'out_file1'
      • Under Rename dataset, and enter a descriptive name for the output dataset like Top 5 exon IDs

    Rename the output of the `Select first` tool step

  6. Repeat this for the output of the Compare two Datasets tool, naming it Top 5 exons

  7. Save your workflow (important!) by clicking on the galaxy-save icon at the top right of the screen.

    Save option in the workflow editor menu

  8. Return to the analysis view by clicking on Analyze Data at the top menu bar.

comment Comments

We could validate our newly built workflow by running it on the same input datasets that we used at the start of this tutorial, in order to make sure we do obtain the same results.

Run workflow on different data

Now that we have built our workflow, let’s use it on some different data. For example, let’s find out which exons have the highest number of repeat elements.

hands_on Hands-on: Run workflow

  1. Create a new history and give it a name.

    tip Tip: Creating a new history

    Click the new-history icon at the top of the history panel

    If the new-history is missing:

    1. Click on the galaxy-gear icon (History options) on the top of the history panel
    2. Select the option Create New from the menu
  2. We will need the list of exons again. We don’t have to get this from UCSC again, we can just copy it from our previous history. The easiest way to do this is to go to the galaxy-columns history overview. Here you can just drag and drop datasets from one history to another.

    Drag and drop of `Exons` dataset in the history overview

  3. Click Analyze Data at the top to return to the main analysis window

  4. We wanted to know something about the repetitive elements per exon. We get this data from UCSC.
    • “assembly”: Dec. 2013 (GRCh38/hg38)
    • “group” parameter should be changed to Repeats
    • “position”: chr22
    • leave the rest of the settings to the defaults

    Click on get output and then Send query to Galaxy on the next screen.

  5. Open the workflow menu (top menu bar). Find the workflow you made in the previous section, and select the option Run.

    `Run` option in the workflow menu

    The central panel will change to allow you to configure and launch the workflow.

  6. Select appropriate datasets for the inputs as shown below, then scroll down and click Run workflow.

    Settings for running the workflow

    comment Potential workflow issues

    Galaxy validates the workflow inputs to ensure they’re correct. It may show a validation error at the start, until you select Exons for the Exons input, and your repeats for the Features input.

  7. Once the workflow has started, you will initially be able to see all its steps, but the unimportant intermediates will disappear after they complete successfully:

    Datasets appearing in the history

comment Comment

Because most intermediate steps of the workflow were hidden, once it is finished you will only see the final two datasets. If we want to view the intermediate files after all, we can unhide all hidden datasets by clicking on “hidden”, just below the history name, or by selecting Unhide Hidden Datasets from the galaxy-gear history options menu.

question Questions

Which exon had the highest number of repeats? How many repeats were there?

Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

To share a history, click on the galaxy-gear icon in the history panel and select Share or Publish. On this page you can do 3 things:

  1. Make History Accessible via Link. This generates a link that you can give out to others. Anybody with this link will be able to view your history.
  2. Make History Accessible and Publish. This will not only create a link, but will also publish your history. This means your history will be listed under Shared Data → Histories in the top menu.
  3. Share with a user. This will share the history only with specific users on the Galaxy instance.

comment Permissions

Different servers have different default permission settings. Some servers create all of your datasets completely private to you, while others make them accessible if you know the secret ID.

Be sure to select Also make all objects within the History accessible whenever you make a history accessible via link, otherwise whomever you send your link to might not be able to see your history.

hands_on Hands-on: Share history and workflow

  1. Share one of your histories with your neighbour.
  2. See if you can do the same with your workflow!
  3. Find the history and/or workflow shared by your neighbour. Histories shared with specific users can be accessed by those users in their galaxy-gear history menu under Histories shared with me.

Conclusion

trophy Well done! You have just performed your first analysis in Galaxy. You also created a workflow from your analysis so you can easily repeat the exact same analysis on other datasets. Additionally you shared your results and methods with others.

keypoints Key points

  • Galaxy provides an easy-to-use graphical user interface for often complex command-line tools

  • Galaxy keeps a full record of your analysis in a history

  • Workflows enable you to repeat your analysis on different data

  • Galaxy can connect to external sources for data import and visualization purposes

  • Galaxy provides ways to share your results and methods with others

congratulations Congratulations on successfully completing this tutorial!