name: inverse layout: true class: center, middle, inverse
---
# Docker and Galaxy
Bérénice Batut
Björn Grüning
last_modification
Updated:
purl
PURL
:
gxy.io/GTN:S00011
text-document
Plain-text slides
|
Tip:
press
P
to view the presenter notes |
arrow-keys
Use arrow keys to move between slides
??? Presenter notes contain extra information which might be useful if you intend to use these slides for teaching. Press `P` again to switch presenter notes off Press `C` to create a new window where the same presentation will be displayed. This window is linked to the main window. Changing slides on one will cause the slide to change on the other. Useful when presenting. --- ### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions - Why Docker? What is it? - How to use Docker? - How to integrate Galaxy in Docker to facilitate its deployment? --- ### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives - Docker basics - Galaxy Docker image (usage) - Galaxy Docker (internals) - Galaxy flavours --- ### Preparing Admin Training VMs If you're falling along on Galaxy Admin Training VMs, you will need to setup Docker. You can modify your Ansible setup to configure this as described at [bit.ly/gxadmin-docker-playbook](http://bit.ly/gxadmin-docker-playbook). You will also need to stop Galaxy running on the VM so that you can run Galaxy in a container. ``` sudo supervisorctl stop galaxy ``` ??? TODO: Remove this slide after the training, it isn't general. --- ### Preparation ``` $ docker pull bgruening/galaxy-stable:18.05 $ git clone https://github.com/bgruening/docker-galaxy-stable.git ``` ??? Here you will learn the internals of the Docker Galaxy Image. We will show you tips and tricks on how to run the Galaxy Docker Image successfully in production, how to manage updates and how to bind the container to a cluster scheduler. Moreover, you will learn how to create your own Galaxy flavour mixing a variety of different tools and visualisations --- ## Why Docker? <br>What is it? ![Image of a blue whale with containers on its back.](../../images/docker_whale.png) --- ### Deployment issues of bioinformatics tools - Different environments - Different OS - Different packaging - Conflict between tools or versions <i class="fa fa-long-arrow-right"></i> Impact on usability and reproducibility --- ### Deployment issue ![Galaxy is a box with many dependencies like node, java, nginx, python, uwsgi, anaconda. On the right knime is another box with java and eclipse dependencies. Lines show the multiplicity of tools and dependencies trying to run on multiple environments with windows/osx/linux desktops, servers, and the cloud.](../../images/deployment_issue.svg) --- ### Deployment issue Matrix from Hell ![Galaxy and knime form the y axis of a table with a lot of question marks about which environments support them.](../../images/deployment_issue_matrix.svg) --- ### Transport Pre 1960 ![Icons for multiple types of goods shown multiple methods for transport and storing.](../../images/cargo_transport_issue.svg) --- ### Transport Pre 1960 Matrix from Hell ![A similar table full of question marks between goods and shipment methods is shown.](../../images/cargo_transport_matrix.svg) --- ### Intermodal shipping container ![A blue shipping container is shown solving the transport problem by being a one size container to fit all sorts of goods, on all sorts of transports.](../../images/cargo_transport_solution.svg) ??? A standard container - Virtually loading any goods - Sealed until it reaches final delivery - Can be - loaded and unloaded - stacked - transported efficiently over long distances - transferred from one mode of transport to another --- ### Docker ![Docker logo on a blue shipping container solves the galaxy/knime deployment problem across environments.](../../images/deployment_solution.svg) ??? Docker = An engine that encapsulated any tools as a - lightweight - portable - self sufficient container - manipulated using standard operations - run consistently on virtually any hardware platform --- ### A Docker container? ![A schematic with the kernal below and two containers is shown, each container has different stacks.](../../images/WhatIsDocker_1_kernal-2_1.png) Lightweight and Open ??? - lightweight: same OS kernel, instant start, less RAM use - open: open standard, run on all major unix distributions and windows --- ### Virtual machines vs Containers ![A cartoon comparing containers vs VMs. On the left the VMs are presented as a layer of server, host os, and hypervisor (type 2) followed by several tall pillars of guest OS, bin/libs and finally their respective apps. Everything is siloed. On the right is containers where the layers are server, host os, and docker engine. Above those are very short pillars of each different app, and when duplicate apps are run, the bin/libs are shown as shared. Text reads: containers are isolated but share OS and where appropriate, bins/libraries.](../../images/container_vs_vm.svg) Containers more portable and efficient ??? Similar resource isolation and allocation benefits *but* a different architectural approach - VM: application + necessary binaries and libraries + entire guest operating system - All of which can amount to tens of GBs - Container: application and all of its dependencies - Share the kernel with other containers - Running as isolated processes in user space on the host operating system --- ## How to use Docker? ![A cartoon of a whale with shipping containers atop it.](../../images/docker-usage.jpg) --- ### The client ![A command line is explained, docker is the program you're using, run is the subcommand that creates and runs a container, and hello-world is the image you want to load.](../../images/docker_command.png) ??? Docker Engine = the core Docker technology on your computer Client: to talk with Docker `docker --help` --- ### Containers? Images? ![A single container labelled galaxy with postgres, nginx, and python inside is shown pointing to three other galaxy containers.](../../images/docker_concept_image_container.png) ??? Image: - recipe that tell how to build your container - A filesystem and parameters to use at runtime - snapshot of the content of a container: it doesn't have state and never changes once built Container - a running instance of an image - you can execute several time the same recipe and if you follow every time the same recipe, you will have the same meal several time. Same for containers --- ### How to get images? ![Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.](../../images/docker_concept_pull.png) ??? Docker Registry: - stateless - highly scalable server side application to stores and distribute Docker images Example: docker hub, quay.io --- ### `docker pull` ```sh $ docker pull hello-world Using default tag: latest latest: Pulling from library/hello-world c04b14da8d14: Pull complete Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9 Status: Downloaded newer image for hello-world:latest ``` --- ### Creation of containers ![Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.](../../images/docker_concept_run.png) --- ### `docker run` ```sh $ docker run hello-world Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker Hub account: https://hub.docker.com For more examples and ideas, visit: https://docs.docker.com/engine/userguide/ ``` --- ### `docker run` ```sh $ docker run docker/whalesay cowsay Galaxy Unable to find image 'docker/whalesay:latest' locally latest: Pulling from docker/whalesay e190868d63f8: Pull complete 909cd34c6fd7: Pull complete 0b9bfabab7c1: Pull complete a3ed95caeb02: Pull complete 00bf65475aba: Pull complete c57b6bcc83e3: Pull complete 8978f6879e2f: Pull complete 8eed3712d2cf: Pull complete Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b Status: Downloaded newer image for docker/whalesay:latest ________ < Galaxy > -------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ ``` ??? - Automatic `pull` if image not findable - Interaction with the container to say something --- ### `docker run` ```sh $ docker run --help Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...] Run a command in a new container -a, --attach=[] Attach to STDIN, STDOUT or STDERR --add-host=[] Add a custom host-to-IP mapping (host:ip) --cpu-shares CPU shares (relative weight) ... -d, --detach Run container in background and print container ID -e, --env=[] Set environment variables --entrypoint Overwrite the default ENTRYPOINT of the image --env-file=[] Read in a file of environment variables ... -h, --hostname Container host name -i, --interactive Keep STDIN open even if not attached --name Assign a name to the container --net=default Connect a container to a network ... -P, --publish-all Publish all exposed ports to random ports -p, --publish=[] Publish a container's port(s) to the host --privileged Give extended privileges to this container --rm Automatically remove the container when it exits -t, --tty Allocate a pseudo-TTY -v, --volume=[] Bind mount a volume ... ``` --- ### Run an interactive container ```sh $ docker run -t -i docker/whalesay root@7de97f8dd5eb:/cowsay# root@7de97f8dd5eb:/cowsay# cowsay Galaxy ________ < Galaxy > -------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ root@7de97f8dd5eb:/cowsay# ``` --- ### Control during startup ``` $ docker run -i -t -p 8080:80 -p 8021:21 -p 9002:9002 --privileged=true -e "NONUSE=reports" -e "GALAXY_CONFIG_ADMIN_USERS=albert@einstein.gov" -e "GALAXY_CONFIG_MASTER_API_KEY=83D4jaba7330aDKHkakjGa937" -e "GALAXY_CONFIG_BRAND='My own Galaxy flavour'" -e "GALAXY_LOGGING=full" bgruening/galaxy-stable:18.05 ``` ??? possibility to add many different parameters to control the container when starting it --- ### Management of data ```sh $ mkdir data $ docker run docker/whalesay cowsay Galaxy > data/cowsay $ more data/cowsay ________ < Galaxy > -------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ $ docker run -t -i docker/whalesay root@f4fa8ed32ef8:/cowsay# ls ChangeLog INSTALL LICENSE MANIFEST README Wrap.pm.diff cows cowsay cowsay.1 install.pl install.sh pgp_public_key.txt root@f4fa8ed32ef8:/cowsay# cowsay Hello Galaxy > cowsay2 ``` Can we access the `cowsay` file inside the container? <br>And the `cowsay2` file outside the container? --- ### Management of data ![A container is shown with an arrow pointing to data and a big x over it.](../../images/docker_volume_closed.png) ??? A container is closed --- ### Management of data ![The same as before, but inside the container it wants to create a new file in that same folder but it cannot.](../../images/docker_volume_open.png) ??? A container is closed --- ### Data volume ![A data volume with two files is shown as a folder, now this is moutned inside two containers.](../../images/docker_volume.png) ??? Data volume - a specially-designated directory - within one or more containers - bypasses the Union File System Data volumes designed to persist data, independent of the container’s life cycle: - Initialisation when a container is created - Possibility of sharing and reusing among containers - Direct changes to a data volume - Persistence of data volumes even if the container itself is deleted --- ### Data volume ![Another command line explained, now docker run -v host_dir:container_dir ..., where host_dir is the absolute path or data volume name, and container_dir is where it should be mounted.](../../images/docker_run_volume.png) ??? How to get a data volume? --- ### Data volume ```sh $ ls data/ cowsay_Galaxy $ docker run -t -i -v $PWD/data:/data docker/whalesay root@f4fa8ed32ef8:/cowsay# ls /data cowsay_Galaxy root@f4fa8ed32ef8:/cowsay# cowsay Galaxy2 > /data/cowsay_Galaxy2 root@f4fa8ed32ef8:/cowsay# ls /data cowsay_Galaxy cowsay_Galaxy2 root@f4fa8ed32ef8:/cowsay# exit $ ls data/ cowsay_Galaxy cowsay_Galaxy2 ``` --- ### Execution of commands <br>inside a running container ![The schematic of a docker host with three galaxies is running, docker exec points to one of the galaxy containers.](../../images/docker_concept_exec.png) ??? Run a command in a running container --- ### `docker exec` ```sh $ docker run -d docker/whalesay /bin/sh -c "while true; do sleep 1; done" 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 $ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \ cowsay Galaxy ________ < Galaxy > -------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ ``` --- ### Stop/Start containers ![Again the same graphic, but now one of the galaxy containers becomes stopped through the docker stop command and turns grey. Docker start brings it back to running.](../../images/docker_concept_start_stop.png) Note: Sending SIGTERM and then SIGKILL after a grace period --- ### `docker stop` & `docker start` ```sh $ docker stop 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 $ $ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \ cowsay Galaxy Error response from daemon: Container 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \ is not running $ $ docker start 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 $ $ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \ cowsay Galaxy ________ < Galaxy > -------- \ \ \ ## . ## ## ## == ## ## ## ## === /""""""""""""""""___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/ ``` --- ### View all containers ![docker ps shows a box around the previous graphic and shows running and stopped container.](../../images/docker_concept_ps.png) --- ### `docker ps` ```sh $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7179e85085ef docker/whalesay "/bin/sh -c 'while tr" 12 minutes ago Up 2 seconds agitated_lovelace $ $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7de97f8dd5eb docker/whalesay "/bin/bash" 4 minutes ago Exited (0) About a minute ago trusting_swanson 9218bbee9c48 docker/whalesay "/bin/bash" 5 minutes ago Exited (0) 4 minutes ago evil_swirles 7179e85085ef docker/whalesay "/bin/sh -c 'while tr" 13 minutes ago Up 55 seconds agitated_lovelace ad275579c454 ubuntu "/bin/sh -c 'while tr" 15 minutes ago Exited (137) 13 minutes ago condescending_mestorf 66179c4d16da ubuntu "/bin/bash" About an hour ago Exited (130) 15 minutes ago determined_pasteur 27386c8b69b3 ubuntu "/bin/sh" About an hour ago Exited (0) About an hour ago lonely_ramanujan 4cfefa19e6fa docker/whalesay "/bin/bash" About an hour ago Exited (0) About an hour ago thirsty_chandrasekhar 82687eb94ab9 docker/whalesay "cowsay Galaxy" 2 hours ago Exited (0) 2 hours ago fervent_babbage 6dbabb9384ad tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Exited (0) 7 days ago tender_bhaskara 5d6f09b94727 tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Exited (0) 8 days ago jolly_brattain 4e6f38b4c34c tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Created angry_colden b3e6c7412a75 tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Created desperate_visvesvaraya 1ec56c9e37f8 tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Created hopeful_khorana 2b129d00eb10 tmp-bioconda-builder "/usr/local/bin/tini " 8 days ago Created gigantic_ptolemy da45ab698f58 fb77c13d04c0 "/usr/local/bin/tini " 13 days ago Exited (0) 13 days ago jovial_yalow 48dc3ed4e173 fb77c13d04c0 "/usr/local/bin/tini " 13 days ago Created focused_ritchie e9195b6512dd a2107450fdf2 "/usr/local/bin/tini " 2 weeks ago Created thirsty_bardeen ``` Note: Name of containers and id --- ### Creation of a new image ![A dockerfile built with docker build builds a new container on the local docker host.](../../images/docker_concept_build.png) ??? Need a Dockerfile. I will describe it later. Here it is just to give you a broad overview of the principles --- ### View all images ![Docker image shows a list of all the images on the docker host that can be run.](../../images/docker_concept_image.png) --- ### Push your image on a registry ![Docker push is shown pushing the locally built container to the image registry.](../../images/docker_concept_push.png) --- <hands-on-title>Hands on! (1)</hands-on-title> Play with `quay.io/biocontainers/samtools:1.3--1` 1. Get the image 2. Launch `samtools` container interactively 1. Print the help page for `samtools` 3. Launch a `samtools` container in detach mode 1. Check if it exists and find its name 2. Stop it and restart it 3. Print the help page using this container --- <hands-on-title>Hands on! (2)</hands-on-title> Play with `quay.io/biocontainers/samtools:1.3--1` 1. Create a local `samtool_dir` directory 2. Launch an interactive container with a volume pointing at the `samtool_dir` directory 1. Add a [toy sam file](https://raw.githubusercontent.com/samtools/samtools/develop/examples/toy.sam) into the local `samtool_dir` directory 2. Check if the existence of the file from inside the container 3. Visualize the content of the toy file with `samtools view` commands 4. Generate stats of the toy file into a `toy_stat` file into the local `samtool_dir` directory ??? data volume is the way to make your tool connecting --- ## How to containerize your tools? ![Cartoon of a docker container](../../images/docker-container.jpg) ??? Now I am sure all of you want to containerize his tool :) --- ### How to create your image? ![Cartoon showing a dockerfile building a container with docker build.](../../images/docker_build.svg) --- ### A Dockerfile? A text document that contains all the commands a user could call on the command line to assemble an image ```sh # Comment INSTRUCTION arguments ``` .footnote[[Best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)] ??? The paper describing the recipe for you container Review of the commands --- ### Dockerfile & Layers ![dockerfile with steps like run and add and set user and command are shown inverted as the layers of an image, each building on the last.](../../images/dockerfile_layers.svg) --- ### But Don't Do it Manually, Use Biocontainers [Tool Dependencies and Containers](/training-material/topics/dev/tutorials/containers/slides.html) --- ## How to integrate Galaxy inside Docker Container? --- ### Galaxy Docker Container A Docker launching a Galaxy instance and - FTP-Server - Webserver - Scheduler - Process control UI - ToolShed ready - Interactive Environment ready ``` docker run -i -t -p 8080:80 bgruening/galaxy-stable:18.05 ``` Visit your machine on port 8080 to see Galaxy running. https://github.com/bgruening/docker-galaxy-stable --- <hands-on-title>Hands on!</hands-on-title> Launch a Galaxy Docker Container and try to - Add Data - Become an Admin - Restart Galaxy - Use export mounts (persistent data) - Trouble shooting (logging) - Install tools - ... --- ### Galaxy + SLURM = <i class="fa fa-heart" aria-hidden="true"></i> https://github.com/bgruening/docker-galaxy-stable/tree/dev/docs/Running_jobs_outside_of_the_container.md --- ### Docker flavours ![Schematic of multiple docker containers and how they layer upon one another. ubuntu forms the base, and requirements builds on that, then galaxy-stable on top of that.](../../images/rna-workbench-chart.png) --- ### Galactic Flavours - [NCBI-Blast](https://github.com/bgruening/docker-galaxy-blast) - [ChemicalToolBox](https://github.com/bgruening/docker-galaxy-chemicaltoolbox) - [ballaxy](https://github.com/anhi/docker-scripts/tree/master/ballaxy) - [NGS-deepTools](https://github.com/bgruening/docker-recipes/blob/master/galaxy-deeptools) - [Galaxy ChIP-exo](https://github.com/gregvonkuster/docker-galaxy-ChIP-exo) - [Galaxy Proteomics](https://github.com/bgruening/docker-galaxyp) - [Imaging](https://github.com/bgruening/docker-galaxy-imaging) - [Constructive Solid Geometry](https://github.com/gregvonkuster/docker-galaxy-csg) - [Galaxy for metagenomics](https://github.com/bgruening/galaxy-metagenomics) - [Galaxy with the Language Application Grid tools](https://github.com/lappsgrid-incubator) - [RNAcommender](https://github.com/gianlucacorrado/galaxy-RNAcommender) - [OpenMoleculeGenerator](https://github.com/bgruening/galaxy-open-molecule-generator) --- ### Create your own Flavour ``` # Galaxy - My own flavour # # VERSION 0.1 FROM bgruening/galaxy-stable:18.05 MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com ENV GALAXY_CONFIG_BRAND deepTools # Adding the tool definitions to the container ADD my_tool_list.yml $GALAXY_ROOT/my_tool_list.yml # Install deepTools RUN install-tools $GALAXY_ROOT/my_tool_list.yml ``` --- ### The tools file ``` galaxy_instance: http://localhost:8080 tools: - name: fastqc owner: devteam tool_panel_section_id: cshl_library_information tool_shed_url: https://toolshed.g2.bx.psu.edu #revisions: ``` --- <hands-on-title>Hands on!</hands-on-title> Create a Galaxy Flavour --- ### Galaxy Flavor Generator ![Website used to build custom flavours of galaxy docker containers](../../images/galaxy_flavour_generator.png) https://github.com/galaxyFlavorsGenerators/galaxyFlavorGenerator --- ### <i class="fas fa-key" aria-hidden="true"></i><span class="visually-hidden">keypoints</span> Key points - Use Docker - Integrate your tools into Galaxy - Use Galaxy inside Docker to have a flavoured Galaxy instance --- ## Thank You! This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!
Authors:
Bérénice Batut
Björn Grüning
Tutorial Content is licensed under
Creative Commons Attribution 4.0 International License
.