Galaxy Monitoring with Reports

Overview
Questions:
  • How to monitor a Galaxy service with the Reports application?

Objectives:
  • Setup and start the Galaxy reports app.

Requirements:
Time estimation: 30 minutes
Supporting Materials:
Last modification: Nov 7, 2022
License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License The GTN Framework is licensed under MIT

The reports application gives some pre-configured analytics screens. These are very easy to setup and can help with debugging issues in Galaxy.

Agenda
  1. Setting up Reports
Comment: Galaxy Admin Training Path

The yearly Galaxy Admin Training follows a specific ordering of tutorials. Use this timeline to help keep track of where you are in Galaxy Admin Training.

  1. Step 1 ansible-galaxy
  2. Step 2 tus
  3. Step 3 cvmfs
  4. Step 4 singularity
  5. Step 5 tool-management
  6. Step 6 data-library
  7. Step 7 connect-to-compute-cluster
  8. Step 8 job-destinations
  9. Step 9 pulsar
  10. Step 10 gxadmin
  11. Step 11 monitoring
  12. Step 12 tiaas
  13. Step 13 reports
  14. Step 14 ftp

Setting up Reports

The reports application is included with the Galaxy codebase and this tutorial assumes you’ve already done all of the setup required for Galaxy, systemd, uWSGI, and NGINX.

Hands-on: Setup Reports
  1. First we add a basic configuration of the Reports app to the playbook templates. Create templates/galaxy/config/ folder, if it doesn’t exist, and create templates/galaxy/config/reports.yml with the following contents:

    --- /dev/null
    +++ b/templates/galaxy/config/reports.yml
    @@ -0,0 +1,26 @@
    +uwsgi:
    +    socket: 127.0.0.1:9001
    +    buffer-size: 16384
    +    processes: 1
    +    threads: 4
    +    offload-threads: 2
    +    static-map: /static/style={{ galaxy_server_dir }}/static/style/blue
    +    static-map: /static={{ galaxy_server_dir }}/static
    +    static-map: /favicon.ico=static/favicon.ico
    +    master: true
    +    virtualenv: {{ galaxy_venv_dir }}
    +    pythonpath: {{ galaxy_server_dir }}/lib
    +    mount: /reports=galaxy.webapps.reports.buildapp:uwsgi_app()
    +    manage-script-name: true
    +    thunder-lock: false
    +    die-on-term: true
    +    hook-master-start: unix_signal:2 gracefully_kill_them_all
    +    hook-master-start: unix_signal:15 gracefully_kill_them_all
    +    py-call-osafterfork: true
    +    enable-threads: true
    +reports:
    +    cookie-path: /reports
    +    database_connection: "{{ galaxy_config.galaxy.database_connection }}"
    +    file_path: /data
    +    filter-with: proxy-prefix
    +    template_cache_path: "{{ galaxy_mutable_data_dir }}/compiled_templates"
       
    

    If you haven’t worked with diffs before, this can be something quite new or different.

    If we have two files, let’s say a grocery list, in two files. We’ll call them ‘a’ and ‘b’.

    Old
    $ cat old
    🍎
    🍐
    🍊
    🍋
    🍒
    🥑
    New
    $ cat new
    🍎
    🍐
    🍊
    🍋
    🍍
    🥑

    We can see that they have some different entries. We’ve removed 🍒 because they’re awful, and replaced them with an 🍍

    Diff lets us compare these files

    $ diff old new
    5c5
    < 🍒
    ---
    > 🍍

    Here we see that 🍒 is only in a, and 🍍 is only in b. But otherwise the files are identical.

    There are a couple different formats to diffs, one is the ‘unified diff’

    $ diff -U2 old new
    --- old 2022-02-16 14:06:19.697132568 +0100
    +++ new 2022-02-16 14:06:36.340962616 +0100
    @@ -3,4 +3,4 @@
    🍊
    🍋
    -🍒
    +🍍
    🥑

    This is basically what you see in the training materials which gives you a lot of context about the changes:

    • --- old is the ‘old’ file in our view
    • +++ new is the ‘new’ file
    • @@ these lines tell us where the change occurs and how many lines are added or removed.
    • Lines starting with a - are removed from our ‘new’ file
    • Lines with a + have been added.

    So when you go to apply these diffs to your files in the training:

    1. Ignore the header
    2. Remove lines starting with - from your file
    3. Add lines starting with + to your file

    The other lines (🍊/🍋 and 🥑) above just provide “context”, they help you know where a change belongs in a file, but should not be edited when you’re making the above change. Given the above diff, you would find a line with a 🍒, and replace it with a 🍍

    Added & Removed Lines

    Removals are very easy to spot, we just have removed lines

    --- old	2022-02-16 14:06:19.697132568 +0100
    +++ new 2022-02-16 14:10:14.370722802 +0100
    @@ -4,3 +4,2 @@
    🍋
    🍒
    -🥑

    And additions likewise are very easy, just add a new line, between the other lines in your file.

    --- old	2022-02-16 14:06:19.697132568 +0100
    +++ new 2022-02-16 14:11:11.422135393 +0100
    @@ -1,3 +1,4 @@
    🍎
    +🍍
    🍐
    🍊

    Completely new files

    Completely new files look a bit different, there the “old” file is /dev/null, the empty file in a Linux machine.

    $ diff -U2 /dev/null old
    --- /dev/null 2022-02-15 11:47:16.100000270 +0100
    +++ old 2022-02-16 14:06:19.697132568 +0100
    @@ -0,0 +1,6 @@
    +🍎
    +🍐
    +🍊
    +🍋
    +🍒
    +🥑

    And removed files are similar, except with the new file being /dev/null

    --- old	2022-02-16 14:06:19.697132568 +0100
    +++ /dev/null 2022-02-15 11:47:16.100000270 +0100
    @@ -1,6 +0,0 @@
    -🍎
    -🍐
    -🍊
    -🍋
    -🍒
    -🥑
  2. In your galaxyservers group variables file, tell the playbook to deploy the reports configuration file:

    --- a/group_vars/galaxyservers.yml
    +++ b/group_vars/galaxyservers.yml
    @@ -51,6 +51,7 @@ galaxy_root: /srv/galaxy
     galaxy_user: {name: galaxy, shell: /bin/bash}
     galaxy_commit_id: release_22.05
     galaxy_force_checkout: true
    +galaxy_reports_path: "{{ galaxy_config_dir }}/reports.yml"
     miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
     miniconda_version: 4.7.12
     miniconda_manage_dependencies: false
    @@ -131,6 +132,8 @@ galaxy_config_templates:
         dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
       - src: templates/galaxy/config/tool_destinations.yml
         dest: "{{ galaxy_config.galaxy.tool_destinations_config_file }}"
    +  - src: templates/galaxy/config/reports.yml
    +    dest: "{{ galaxy_reports_path }}"
        
     galaxy_local_tools:
     - testing.xml
       
    
  3. Similar to Galaxy we will again use systemd to manage the Reports process.

    --- a/group_vars/galaxyservers.yml
    +++ b/group_vars/galaxyservers.yml
    @@ -144,6 +144,7 @@ galaxy_dynamic_job_rules:
        
     # systemd
     galaxy_manage_systemd: true
    +galaxy_manage_systemd_reports: yes
     galaxy_systemd_env: [DRMAA_LIBRARY_PATH="/usr/lib/slurm-drmaa/lib/libdrmaa.so.1"]
        
     # Certbot
       
    
  4. Then we need to tell NGINX it should serve our Reports app under <server_url>/reports url. Edit your templates/nginx/galaxy.j2 file, and within the server block, add a block for proxying the reports application. It should look like:

    --- a/templates/nginx/galaxy.j2
    +++ b/templates/nginx/galaxy.j2
    @@ -91,4 +91,10 @@ server {
         }
        
         {{ tiaas_nginx_routes }}
    +
    +    location /reports/ {
    +        uwsgi_pass           127.0.0.1:9001;
    +        uwsgi_param          UWSGI_SCHEME $scheme;
    +        include              uwsgi_params;
    +    }
     }
       
    
  5. Run the playbook:

    Input: Bash
    ansible-playbook galaxy.yml
    
  6. The reports application should be available, under <server_url>/reports/.>

Comment: Insecure!

But notice that your Reports server is not secured! Check out the External Authentication tutorial for information on securing Reports.

Comment: Got lost along the way?

If you missed any steps, you can compare against the reference files, or see what changed since the previous tutorial.

If you’re using git to track your progress, remember to add your changes and commit with a good commit message!

Key points
  • Galaxy supports pluggable monitoring extensions.

  • The Reports webapp is one option to monitor your system.

Frequently Asked Questions

Have questions about this tutorial? Check out the tutorial FAQ page or the FAQ page for the Galaxy Server administration topic to see if your question is listed there. If not, please ask your question on the GTN Gitter Channel or the Galaxy Help Forum

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.
Did you use this material as a learner or student? Click the form below to leave feedback.

Click here to load Google feedback frame

Citing this Tutorial

  1. Nate Coraor, Björn Grüning, Simon Gladman, Helena Rasche, Galaxy Monitoring with Reports (Galaxy Training Materials). https://training.galaxyproject.org/training-material/topics/admin/tutorials/reports/tutorial.html Online; accessed TODAY
  2. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012



@misc{admin-reports,
author = "Nate Coraor and Björn Grüning and Simon Gladman and Helena Rasche",
title = "Galaxy Monitoring with Reports (Galaxy Training Materials)",
year = "",
month = "",
day = ""
url = "\url{https://training.galaxyproject.org/training-material/topics/admin/tutorials/reports/tutorial.html}",
note = "[Online; accessed TODAY]"
}
@article{Hiltemann_2023,
	doi = {10.1371/journal.pcbi.1010752},
	url = {https://doi.org/10.1371%2Fjournal.pcbi.1010752},
	year = 2023,
	month = {jan},
	publisher = {Public Library of Science ({PLoS})},
	volume = {19},
	number = {1},
	pages = {e1010752},
	author = {Saskia Hiltemann and Helena Rasche and Simon Gladman and Hans-Rudolf Hotz and Delphine Larivi{\`{e}}re and Daniel Blankenberg and Pratik D. Jagtap and Thomas Wollmann and Anthony Bretaudeau and Nadia Gou{\'{e}} and Timothy J. Griffin and Coline Royaux and Yvan Le Bras and Subina Mehta and Anna Syme and Frederik Coppens and Bert Droesbeke and Nicola Soranzo and Wendi Bacon and Fotis Psomopoulos and Crist{\'{o}}bal Gallardo-Alba and John Davis and Melanie Christine Föll and Matthias Fahrner and Maria A. Doyle and Beatriz Serrano-Solano and Anne Claire Fouilloux and Peter van Heusden and Wolfgang Maier and Dave Clements and Florian Heyl and Björn Grüning and B{\'{e}}r{\'{e}}nice Batut and},
	editor = {Francis Ouellette},
	title = {Galaxy Training: A powerful framework for teaching!},
	journal = {PLoS Comput Biol} Computational Biology}
}

                   

Congratulations on successfully completing this tutorial!