Architecture 08 - Galaxy Application Components: Models, Managers, and Services

Contributors

John Chilton

Questions

• How is business logic organized in Galaxy?

• What are models, managers, and services?

• How does the database layer work?

Objectives

• Understand the three-layer architecture

• Learn about SQLAlchemy and the ORM

• Understand database migrations with Alembic

• Navigate the Galaxy data model

layout: introduction_slides topic_name: Galaxy Architecture

Architecture 08 - Galaxy Application Components: Models, Managers, and Services

The architecture surrounding Galaxy components.

layout: true left-aligned class: left, middle — layout: true class: center, middle

class: center

There are many ways to describe and visualize the Galaxy server architecture, one is to imagine the Galaxy database as the ultimate source for Galaxy “stuff” and the API controllers as the ultimate sink.

In this architecture imagining of Galaxy, managers are the layer meant to mediate all controller interactions (and isolate the backend from the web framework) while the model layer is meant to mediate all database interactions (and isolate the backend from database internals).

class: center

class: enlarge150

Services

Handle API and web processing details of requests and responses at a high-level.

Thin layer below the controllers to shield applciation logic from FastAPI internals.

In practice, it is totally fine to skip this layer and have FastAPI controllers talk directly to managers. Also in practice, there are many places where the controller or service layers are thicker than they should be - and these are anti-patterns that shouldn’t be followed.

class: enlarge150

Managers

High-level business logic that ties all of these components together.

Controllers should ideally be thin wrappers around actions defined in managers.

Whenever a model requires more than just the database, the operation should be defined in a manager instead of in the model.

class: center

Managers - Some Key Files

class: center

Managers - Some Helpers

Galaxy Models

• Database interactions powered by SQLAlchemy - https://www.sqlalchemy.org/.
• Galaxy doesn’t think in terms of “rows” but “objects”.
• Classes for Galaxy model objects defined in lib/galaxy/model/__init__.py.
• Classes mapped to database objects in same module via “declarative mapping”.
  • Classes/attributes mapped to tables/columns
  • Associations between classes mapped to relationships between tables

class: center

Galaxy Database Schema Migrations

• Automated execution of incremental, reversible changes to the database schema.
• Performed to update or revert the database schema to a newer or older version.
• Powered by Alembic - https://alembic.sqlalchemy.org/.
  • (as of 22.05; prior to that by SQLAlchemy Migrate)
• Each file in lib/galaxy/model/migrations/alembic/versions_gxy represents a migration description
  • e7b6dcb09efd_create_gxy_branch.py
  • 6a67bf27e6a6_deferred_data_tables.py
  • b182f655505f_add_workflow_source_metadata_column.py

More on Schema Migrations

• Great documentation in code README - lib/galaxy/model/migrations/README.md
  • Admin perspective on how to migrate databases forward and revert on problems.
  • Developer persepctive on how to add new revisions.
• Galaxy’s data model is split into the galaxy model and the legacy install model:
  • Persisted in one combined database or two separate databases
  • Represented by 2 migration branches: “gxy” and “tsi”
• Schema changes defined in revision modules:
  • lib/galaxy/model/migrations/alembic/versions_gxy (gxy branch: galaxy model)
  • lib/galaxy/model/migrations/alembic/versions_tsi (tsi branch: legacy install model)

class: center

Database Diagram

https://galaxyproject.org/admin/internals/data-model/

class: center

class: center

Dataset Metadata

• Typed key-value pairs attached to HDA.
• Keys and types defined at the datatype level.
• Can be used by tools to dynamically control the tool form.

Speaker Notes

class: center

class: center

class: center

class: center

class: center

Key Points

• Services handle high-level API processing
• Managers contain business logic
• Models mediate database interactions
• SQLAlchemy provides ORM
• Alembic handles schema migrations

Thank you!