View markdown source on GitHub

Running Jobs on Remote Resources with Pulsar





last_modification Published: Jan 1, 1970
last_modification Last Updated: Apr 13, 2024

What are heterogenous compute resources?

Differences in:

Galaxy expects:

Example - Australia


Partial solution - CLI job runner

SSH to remote, submit jobs with CLI sbatch, qsub, etc.

Still depends on shared FS



Galaxy’s remote job management system

Pulsar - Architecture

Pulsar - Architecture


Pulsar Transports - RESTful

Pulsar server listens over HTTP(S)

Pulsar client (Galaxy) initiates connections to Pulsar server

Good for:

Pulsar Transports - AMQP

Pulsar server and client connect to AMQP server

Good for:

Pulsar Transports - Embedded

Galaxy runs Pulsar server internally

Good for:

Pulsar - Job file staging

Pulsar can be configured to push or pull when using RESTful:

Pulsar can use libcurl for more robust transfers with resume capability

AMQP is pull-only because Pulsar does not run HTTP server

Pulsar - Dependency management

Pulsar does not provide Tool Shed tool dependency management. But:

Pulsar - Job management

Pulsar “managers” provide job running interfaces:

Pulsar Australia



Key Points

Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo Tutorial Content is licensed under Creative Commons Attribution 4.0 International License.