🖼️ Galaxy Administrator Time Burden and Technology Usage
##Context
Published: 2024-07-10T18:44:14+00:00Tags: admin, deploying, maintenance, survey
Tags: admin, deploying, maintenance, survey
❓ Using data source tools with Pulsar
Data source tools such as UCSC Main will fail if Pulsar is the default destination.
Published: 2024-01-19T19:50:15+00:00Tags: faqs, admin
❓ Debugging Memory Leaks
memray is a great memory profiler for debugging memory issues.
Published: 2023-09-22T09:44:14+00:00Tags: faqs, admin
❓ Galaxy UI is slow
There is a great Tutorial from @mvdbeek which we recommend you follow.
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
❓ Tool missing from Galaxy
First, restart Galaxy and watch the log for lines like:
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
❓ Debugging tool memory errors
Often the tool output contains one of:
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
❓ Debugging tool errors
Tool stdout/stderr is available in UI under “i” icon on history dataset
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
❓ Blank page or no CSS/JavaScript
This generally means that serving of static content is broken:
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
❓ Database Issues
For slow queries, start with EXPLAIN ANALYZE
Published: 2023-05-11T13:25:10+00:00Tags: faqs, admin
📚 Monitoring Galaxy and Pulsar with Sentry
Overview
Published: 2023-04-19T12:33:06+00:00Tags: admin, ansible, git-gat
❓ Mapping Jobs to Specific Storage By User
It is possible to map your jobs to use specific storage backends based on user! If you have e.g. specific user groups that need their data stored separately from other users, for whatever political reasons, then in your dynamic destination you can do something like:
Published: 2023-04-16T09:02:56+00:00Tags: faqs, admin
❓ Failing all jobs from a specific user
This command will let you quickly fail every job from the user ‘service-account’ (replace with your preferred user)
Published: 2023-04-16T09:02:56+00:00Tags: faqs, admin
❓ Library Permission Issues
When running setup-data-libraries it imports the library with the permissions of the admin user, rather locked down to the account that handled the importing.
Published: 2023-04-16T09:02:56+00:00Tags: faqs, admin
❓ Using Git With Ansible Vaults
When looking at git log to see what you changed, you cannot easily look into
Published: 2023-04-16T09:02:56+00:00Tags: faqs, admin
❓ Opening a split screen in byobu
Opening a split screen in byobu
Published: 2023-04-16T09:02:56+00:00Tags: faqs, admin
❓ How do I see what variables are set for a host?
Published: 2023-04-16T09:02:56+00:00
Tags: faqs, admin
📚 Deploying a Beacon v1 in Galaxy
This tutorial will guide you through setting up a GA4GH Beacon!
Published: 2023-04-16T09:02:56+00:00Tags: admin, ga4gh, beacon, git-gat
📚 Customizing the look of Galaxy (Manual)
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
Published: 2023-04-16T09:02:56+00:00Tags: admin
📚 Customizing the look of Galaxy
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
Published: 2023-04-16T09:02:56+00:00Tags: admin, git-gat
📚 Server Maintenance: Cleanup, Backup, and Restoration
Keeping your Galaxy cleaned up is an important way to retain space, especially since for many groups that is the
Published: 2023-04-16T09:02:56+00:00Tags: admin, ansible, deploying, git-gat
📚 Setting up Celery Workers for Galaxy
Overview
Published: 2023-04-16T09:02:56+00:00Tags: admin, ansible, git-gat
📚 Deploying Tailscale/Headscale for private mesh networking
Tailscale makes secure networking easy, it really is like magic. If you’ve used wireguard before, you know it takes a bit to setup and some configuration if you need to do anything fancy.
Published: 2022-09-21T14:40:57+00:00Tags: admin, wireguard, networking
📚 Deploying Wireguard for private mesh networking
In this tutorial we will briefly cover what Wireguard is and how you can leverage it for your needs. This will not make you an expert on Wireguard but will give you the tools you need in order to setup a local Wireguard network.
Published: 2022-09-21T08:23:59+00:00Tags: admin, wireguard, networking
🖼️ Galaxy and Celery
Can you eat it?
Published: 2022-07-16T17:42:54+00:00Tags: admin, ansible, git-gat
📚 How I learned to stop worrying and love the systemd
Many linux sysadmins with years and years of experience bemoan systemd (“it’s infecting everything! Now it wants to mess with time? And DNS???”) and journalctl (“unix was supposed to be about files!”) and while those are fair complaints and make systemd and friends wildly more opaque than traditional SysV and logging to files, there are some benefits that can be obtained, and may be interesting even to the wise old admins. There is a lot of convenience in systemd that can make the tradeoffs worth it.
Published: 2022-07-06T17:13:50+00:00Tags: admin, ansible, systemd
❓ How many mules?
Start with 2 and add more as needed. If you notice that your jobs seem to inexplicably sit for a long time before being dispatched to the cluster, or after they have finished on the cluster, you may need additional handlers.
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ Is YAML sensitive to True/true/False/false
By this reference, YAML doesn’t really care:
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ How do I know what I can do with a role? What variables are available?
You don’t. There is no standard way for reporting this, but well written roles by trusted authors (e.g. geerlingguy, galaxyproject) do it properly and write all of the variables in the README file of the repository. We try to pick sensible roles for you in this course, but, in real life it may not be that simple.
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ What is the difference between the roles with `role:` prefix and without?
The bare role name is just simplified syntax for the roles, you could equally specifiy role:
Tags: faqs, admin
❓ Error: "skipping: no hosts matched"
There can be multiple reasons this happens, so we’ll step through all of them.
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ What if you forget `--diff`?
If you forget to use --diff, it is not easy to see what has changed. Some modules like the copy and template modules have a backup option. If you set this option, then it will keep a backup copy next to the destination file.
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ Variable connection
When the playbook runs, as part of the setup, it collects any variables that are set. For a playbook affecting a group of hosts named my_hosts, it checks many different places for variables, including “group_vars/my_hosts.yml”. If there are variables there, they’re added to the collection of current variables. It also checks “group_vars/all.yml” (for the built-in host group all). There is a precedence order, but then these variables are available for roles and tasks to consume.
Published: 2022-06-07T12:09:49+00:00Tags: faqs, admin
❓ How to read a Diff
If you haven’t worked with diffs before, this can be something quite new or different.
Published: 2022-03-01T10:29:25+00:00Tags: faqs, admin
📚 Performant Uploads with TUS
Here you’ll learn to setup TUS an open source resumable file upload server to process uploads for Galaxy. We use an external process here to offload the main Galaxy processes for more important work and not impact the entire system during periods of heavy uploading.
Published: 2022-03-01T10:29:25+00:00Tags: admin
📚 Enable upload via FTP
This tutorial will guide you to setup an File Transfer Protocol (FTP) server so galaxy users can use it to upload large files. Indeed, as written on the galaxy community hub, uploading data directly from the browser can be unreliable and cumbersome. FTP will allow users to monitor the upload status as well as resume interrupted transfers.
Published: 2021-06-18T13:52:44+00:00Tags: admin, data, git-gat
❓ Running Ansible on your remote machine
It is possible to have ansible installed on the remote machine and run it there, not just from your local machine connecting to the remote machine.
Published: 2021-03-24T13:50:23+00:00Tags: faqs, admin
❓ Operating system compatibility
These Ansible roles and training materials were last tested on Centos 7 and Ubuntu 18.04, but will probably work on other RHEL and Debian variants.
Published: 2021-03-24T13:50:23+00:00Tags: faqs, admin
📰 Galaxy Admin Training Success
The Galaxy Admins and the GTN instructors community has just run the largest-ever Galaxy Admin training event.
Published: 2021-03-01T00:00:00+00:00Tags: news, admin, gat, event, pandemic, remote-teaching
📚 Mapping Jobs to Destinations using TPV
This tutorial heavily builds on the Connecting Galaxy to a compute cluster and it’s expected you have completed this tutorial first.
Published: 2021-01-17T11:37:17+00:00Tags: admin, jobs, git-gat
🖼️ Storage Management
Data Libraries
Published: 2021-01-14T08:41:22+00:00Tags: admin, ansible, storage, git-gat
📚 Data Libraries
Data libraries are a great way to provide structured repositories of data to
Published: 2021-01-14T08:41:22+00:00Tags: admin, ansible, storage, git-gat
📚 Use Apptainer containers for running Galaxy jobs
In this tutorial you will learn how to configure Galaxy to run jobs using Apptainer containers provided by the BioContainers community.
Published: 2021-01-08T14:43:22+00:00Tags: admin, jobs, ansible, git-gat
📚 Reference Data with CVMFS without Ansible
Overview
Published: 2020-06-17T17:37:46+00:00Tags: admin
🖼️ Galaxy Interactive Tools
History
Published: 2020-03-02T18:20:57+00:00Tags: admin, ansible, interactive-tools
🖼️ Running Jobs on Remote Resources with Pulsar
What are heterogenous compute resources?
Published: 2020-02-28T19:24:42+00:00Tags: admin, ansible, jobs, git-gat
📚 Galaxy Interactive Tools
Galaxy Interactive Tools (GxITs) are a method to run containerized tools that are interactive in nature. Interactive Tools typically run a persistent service accessed on a specific port and run until terminated by the user. One common example of such a tool is Jupyter Notebook. Galaxy Interactive Tools are similar in purpose to Galaxy Interactive Environments (GIEs), but are implemented in a significantly different manner. Most notably, instead of directly invoking containers on the Galaxy server, dedicated Docker node, or as a Docker Swarm service (as is done for GIEs), Interactive Tools are submitted through Galaxy’s job management system and thus are scheduled the same as any other Galaxy tool - on a Slurm cluster, for instance. Galaxy Interactive Tools were introduced in Galaxy Release 19.09.
Published: 2020-02-27T15:21:14+00:00Tags: admin, ansible, interactive-tools
📚 Training Infrastructure as a Service (TIaaS)
Galaxy is widely used for teaching. In order to facilitate instructors, the Galaxy Project has developed Training Infrastructure as a Service (TIaaS).
Published: 2020-02-12T14:46:40+00:00Tags: admin, ansible, training, jobs, git-gat
🖼️ Galaxy Installation with Ansible
Install PostgreSQL & Galaxy extensions
Published: 2020-02-04T13:39:23+00:00Tags: admin, ansible, deploying, git-gat
📚 Automation with Jenkins
Overview
Published: 2019-08-06T13:56:16+00:00Tags: admin, ansible, automation
📚 Galaxy Installation on Kubernetes
Galaxy Helm Chart
Published: 2019-07-02T18:24:39+00:00Tags: admin, kubernetes
📚 Managing Galaxy on Kubernetes
Managing Galaxy on Kubernetes
Published: 2019-07-02T18:24:39+00:00Tags: admin, kubernetes
🖼️ Reference Data with CVMFS
Built in Data
Published: 2019-04-26T16:41:13+00:00Tags: admin, ansible, git-gat
🖼️ Galaxy Monitoring with gxadmin
Database Queries
Published: 2019-04-26T16:41:13+00:00Tags: admin, monitoring, ansible, git-gat
📚 Galaxy Monitoring with Reports
The reports application gives some pre-configured analytics screens. These are very easy to setup and can help with debugging issues in Galaxy.
Published: 2019-01-31T15:17:51+00:00Tags: admin, ansible, monitoring, git-gat, broken
🖼️ Galaxy Monitoring with Telegraf and Grafana
Telegraf, InfluxDB, and Grafana
Published: 2019-01-31T15:17:51+00:00Tags: admin, ansible, monitoring, git-gat
📚 Galaxy Monitoring with Telegraf and Grafana
Monitoring is an incredibly important part of server monitoring and maintenance. Being able to observe trends and identify hot spots by collecting metrics gives you a significant ability to respond to any issues that arise in production. Monitoring is quite easy to get started with, it can be as simple as writing a quick shell script in order to start collecting metrics.
Published: 2019-01-31T15:17:51+00:00Tags: admin, ansible, monitoring, git-gat
🖼️ Server Maintenance: Cleanup, Backup, and Restoration
Server Maintenance
Published: 2019-01-31T15:17:51+00:00Tags: admin, ansible, deploying, git-gat
📚 Galaxy Installation with Ansible
This tutorial assumes you have some familiarity with Ansible and are comfortable with writing and running playbooks. If not, please consider following our Ansible Tutorial first.
Published: 2019-01-28T10:40:06+00:00Tags: admin, ansible, deploying, git-gat
📚 Galaxy Monitoring with gxadmin
We will just briefly cover the features available in gxadmin, there are lots of queries that may or may not be useful for your Galaxy instance and you will have to read the documentation before using them.
Published: 2019-01-28T04:20:16+00:00Tags: admin, monitoring, ansible, git-gat
🖼️ Storage Management
Object Store Plugins
Published: 2019-01-28T04:12:36+00:00Tags: admin, ansible, storage
📚 Distributed Object Storage
You may find that your Galaxy files directory has run out of space, but you don’t want to move all of the files from one filesystem to another. One solution to this problem is to use Galaxy’s hierarchical object store to add an additional file space for Galaxy.
Published: 2019-01-28T04:12:36+00:00Tags: admin, ansible, storage
🖼️ Galaxy Troubleshooting
In system administration…
Published: 2019-01-28T04:08:04+00:00Tags: admin
🖼️ Controlling Galaxy with systemd or Supervisor
class: special
Published: 2019-01-28T04:08:04+00:00Tags: admin
🖼️ External Authentication
Auth Mechanisms supported
Published: 2019-01-28T04:08:04+00:00Tags: admin, authentication
📚 Running Jobs on Remote Resources with Pulsar
Pulsar is the Galaxy Project’s remote job running system. It was written by John Chilton (@jmchilton) of the Galaxy Project. It is a python server application that can accept jobs from a Galaxy server, submit them to a local resource and then send the results back to the originating Galaxy server.
Published: 2019-01-28T04:02:36+00:00Tags: admin, ansible, jobs, git-gat
🖼️ Galaxy Tool Management with Ephemeris
Galaxy tools
Published: 2019-01-27T22:09:32+00:00Tags: admin, tools, git-gat
📚 Galaxy Tool Management with Ephemeris
This tutorial will introduce you to one of Galaxy’s associated projects - Ephemeris. Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.
Published: 2019-01-27T22:09:32+00:00Tags: admin, tools, git-gat
🖼️ Terraform
Why Terraform
Published: 2018-10-27T07:51:23+00:00Tags: admin, terraform, deploying, cloud
📚 Deploying a compute cluster in OpenStack via Terraform
Overview
Published: 2018-10-27T07:51:23+00:00Tags: admin, terraform, deploying, cloud
🖼️ Connecting Galaxy to a compute cluster
Galaxy Job Configuration
Published: 2018-01-07T13:46:01+00:00Tags: admin, jobs, ansible, git-gat
📚 Connecting Galaxy to a compute cluster
The tools that are added to Galaxy can have a wide variance in the compute resources that they require and work efficiently on.
Published: 2018-01-07T13:46:01+00:00Tags: admin, jobs, ansible, git-gat
🖼️ User, Role, Group, Quota, and Authentication managment
Users
Published: 2018-01-07T13:28:18+00:00Tags: admin
🖼️ Advanced customisation of a Galaxy instance
Setup
Published: 2017-06-27T19:49:22+00:00Tags: admin
📚 Galaxy Database schema
For the hands-on examples you need access to a Galaxy server and access to its PostgreSQL database. You can set-up this yourself, or use the Galaxy Docker Image provided by Björn Grüning (https://github.com/bgruening/docker-galaxy-stable). During this tutorial, we will work with the Galaxy Docker Image.
Published: 2016-07-20T11:10:47+00:00Tags: admin