Debugging Memory Leaks

memray is a great memory profiler for debugging memory issues.

In the context of Galaxy, this is significantly easier for job handlers. Install it in your virtualenv and

memray run  --trace-python-allocators -o the_dump <your_handler_startup_command_here>

Once you’ve collected enough data,

memray flamegraph --leaks --temporal the_dump -o the_dump.html

would then produce a report that shows allocation made but not freed over time.

It might also be useful to just check what the process is doing with py-spy dump.

You can follow web workers in gunicorn with

memray run --follow-fork -o the_dump gunicorn 'galaxy.webapps.galaxy.fast_factory:factory()' --timeout 600 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b localhost:8082 --config python:galaxy.web_stack.gunicorn_config -w 1 --preload

the traced app will run on port 8082, you can then for instance in an upstream nginx section direct a portion of the traffic to your profiled app.

Persistent URL

Still have questions?