Server Maintenance: Cleanup, Backup, and Restoration

Contributors

Nate Coraor Björn Grüning Simon Gladman Helena Rasche

Questions

• How can I back up my Galaxy?

• What data should be included?

• How can I ensure jobs get cleaned up appropriately?

• How do I maintain a Galaxy server?

• What happens if I lose everything?

Objectives

• Learn about different maintenance steps

• Setup postgres backups

• Setup cleanups

• Learn what to back up and how to recover

Server Maintenance

Speaker Notes

• Server Maintenance

Dataset Cleanup

Can only delete (or purge) dataset when all ‘associations’ pointing at it have been marked deleted.

Speaker Notes

• One of the important admin tasks is to keep an eye on the storage consumption.
• Every instance should have a data rention policy defined and shared with users.
• Codebase contains scripts that can assist with cleaning up and reclaiming space.
• The gxadmin tool can assist with invoking these scripts.

class: reduce70

pgcleanup invocation

$ ./scripts/cleanup_datasets/pgcleanup.py --help
usage: pgcleanup.py [-h] [-c CONFIG_FILE] [-d] [--dry-run] [--force-retry]
                    [-o DAYS] [-U] [-s SEQUENCE] [-w WORK_MEM] [-l LOG_DIR]
                    [ACTION [ACTION ...]]

positional arguments:
  ACTION                Action(s) to perform, chosen from: delete_datasets,
                        delete_exported_histories, delete_inactive_users,
                        delete_userless_histories, purge_datasets,
                        purge_deleted_hdas, purge_deleted_histories,
                        purge_deleted_users, purge_error_hdas,
                        purge_hdas_of_purged_histories,
                        purge_historyless_hdas, update_hda_purged_flag

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE, --config CONFIG_FILE
                        Galaxy config file (defaults to
                        $GALAXY_ROOT/config/galaxy.yml if that file exists or
                        else to ./config/galaxy.ini if that exists). If this
                        isn't set on the command line it can be set with the
                        environment variable GALAXY_CONFIG_FILE.
  -d, --debug           Enable debug logging (SQL queries)
  --dry-run             Dry run (rollback all transactions)
  --force-retry         Retry file removals (on applicable actions)
  -o DAYS, --older-than DAYS
                        Only perform action(s) on objects that have not been
                        updated since the specified number of days
  -U, --no-update-time  Don't set update_time on updated objects
  -s SEQUENCE, --sequence SEQUENCE
                        DEPRECATED: Comma-separated sequence of actions
  -w WORK_MEM, --work-mem WORK_MEM
                        Set PostgreSQL work_mem for this connection
  -l LOG_DIR, --log-dir LOG_DIR
                        Log file directory

Speaker Notes

• Cleanup scripts provide various options for filtering on which datasets to take action.
• Use the dry run option to verify what datasets you are about to affect.

class: reduce70

pgcleanup actions

Speaker Notes

• Some available actions are safer than others, for example “delete userless histories”.
• In Galaxy, when something is marked deleted, it still exists with a deleted flag.
• Once purged the file is gone.

class: reduce70

More pgcleanup actions

Speaker Notes

• Be wary about aggressive storage reclamation. It can aggravate users.
• Well-defined and updated data retention policy brings benefits.

Additional cleaning scripts

• admin_cleanup_datasets.py
  • Mark datasets as deleted that are older than specified cutoff.
  • Has email templated notification (can also be used to just send info without deleting).
  • Can be restricted to a tool.

Speaker Notes

• One of the included scripts can send templated email to the affected users.
• Take advantage of this to give your users some time to backup their data before you reclaim the space.

Other disk space cleanups

• tmpwatch (tmpreaper on Debian-based distros) your job working directory
  • cleanup_job in galaxy.yml (defaults to always though)
• tmpwatch your new_file_path
  • /usr/bin/tmpwatch -v --mtime --dirmtime 7d /srv/galaxy/var/tmp

Speaker Notes

• Galaxy generates intermediate data as part of the job execution.
• These should be automatically cleaned up unless you change the cleanup_job flag in galaxy’s configuration.

Backups

Speaker Notes

• Backups

class: left

What to backup

Must be backed up:

If absolute reproducibility matters:

Speaker Notes

• Many parts of Galaxy are convenient to back up since they are straight folder hierarchies.
• Use your system’s recommended way to back up galaxy’s database.
• If you have local modifications to galaxy’s code back up those as well.

class: left

What not to back up

Restorable from Ansible Playbook:

Recreatable at runtime:

Restoring from backups

If lost database or managed/mutable configs, then restore these first

Then run playbook

Speaker Notes

• In case of an incident do not rush even when pressure is piling.
• Careful planning of the restoration procedure will save you from hasty recovery attempts with errors.
• Consider running a recovery drill pretending that e.g. your galaxy machine was wiped.

Key Points

• Use configuration management (e.g. Ansible)
• Store configuration management in git
• Back up the parts of Galaxy that can't be recreated

Thank you!