You finally decided to analyse some single cell data, you got your files either from the lab or publicly available sources, you opened the first tutorial available on Galaxy Training Network and… you hit the wall! The format of your files is not compatible with the one used in tutorial! Have you been there? This tutorial was created to help you overcome that problem and ensure data interoperability in single cell analysis. Once you get your data into Galaxy in the right format, that’s already 50% of success. Additionally, by using format conversion, you will be able to use different packages presented in tutorials that may require different datatypes.

Agenda

In this tutorial, we will cover:

1. Single cell datatypes
2. Data import
3. SCEasy Tool
4. Seurat -> AnnData
   1. Quick one tool method
   2. Convert to AnnData object compatible with Filter, Plot, Explore workflow
5. AnnData -> Seurat
   1. Quick one step method
   2. Manual conversion
6. AnnData -> SingleCellExperiment (SCE)
   1. Quick single tool method
   2. Convert manually
7. Anndata -> Cell Data Set (CDS)
   1. CDS input for Monocle3 tutorial (use for pre-processed data)
   2. Get data
   3. Extracting annotations
   4. Cell metadata
   5. Gene annotations
   6. Expression matrix
   7. Creating CDS object
   8. General AnnData to CDS conversion
8. Downsampling FASTQ files

Single cell datatypes

To start with, here are the most common formats and datatypes that you might come across if you work with single cell data:

• Tabular - simply using TSV, CSV or TXT formats to store expression matrix as well as cell and gene metadata.
• MTX - it’s just a sparse matrix format with genes on the rows and cells on the columns as output by Cell Ranger.
• HDF5 - Hierarchical Data Format - can store datasets and groups. A dataset is a a multidimensional array of data elements, together with supporting metadata. A group is a structure for organizing objects in an HDF5 file. This format allows for storing both the count matrices and all metadata in a single file rather than having separate features, barcodes and matrix files.
• AnnData objects - anndata is a Python package for handling annotated data matrices. In Galaxy, you’ll see AnnData objects in h5ad format, which is based on the standard HDF5 (h5) format. There are lots of Python tools that work with this format, such as Scanpy, MUON, Cell Oracle, SquidPy, etc.
• Loom - it is simply an HDF5 file that contains specific groups containing the main matrix as well as row and column attributes and can be read by any language supporting HDF5. Loompy has been released as a Python API to interact with loom files, and loomR is its implementation in R.
• Zarr - a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing. The Zarr file format offers powerful compression options, supports multiple data store backends, and can read/write your NumPy arrays.
• Seurat objects - a representation of single-cell expression data for R, in Galaxy you might see them in rdata format.
• Single Cell Experiment (SCE) object - defines a S4 class for storing data from single-cell experiments and provides a more formalized approach towards construction and accession of data. The S4 system is one of R’s systems for object oriented programing. In Galaxy you might see SCE objects in rdata format.
• CellDataSet (CDS) object - the main class used by Monocle to hold single cell expression data. In Galaxy you might see CDS objects in rdata format.

Tip: Using tutorial mode

Tools are frequently updated to new versions. Your Galaxy may have multiple versions of the same tool available. By default, you will be shown the latest version of the tool. This may NOT be the same tool used in the tutorial you are accessing. Furthermore, if you use a newer tool in one step, and try using an older tool in the next step… this may fail! To ensure you use the same tool versions of a given tutorial, use the Tutorial mode feature.

• Open your Galaxy server
• Click on the curriculum icon on the top menu, this will open the GTN inside Galaxy.
• Navigate to your tutorial
• Tool names in tutorials will be blue buttons that open the correct tool for you
• Note: this does not work for all tutorials (yet)
• You can click anywhere in the grey-ed out area outside of the tutorial box to return back to the Galaxy analytical interface

Warning: Not all browsers work!

• We’ve had some issues with Tutorial mode on Safari for Mac users.
• Try a different browser if you aren’t seeing the button.

Data import

As you can see above, there are multiple ways to store single cell data. Therefore, there are also many ways how you can get that data! Obviously, before any format conversion, we need to import the data. In our tutorials we often use Zenodo links, but you can also upload the files directly from your computer. There are also publicly available resources which you can easily access through public atlases, such as Single Cell Expression Atlas or Human Cell Atlas data portal. We created a dedicated tutorial to show how to use those atlases to retrieve data. But today we’re here to focus on data conversion!

SCEasy Tool

In Galaxy Toolshed there is a wonderful tool called SCEasy ( Galaxy version 0.0.7+galaxy2) which allows you to convert between common single cell formats, such as:

• AnnData to CellDataSet (CDS)
• AnnData to Seurat
• Loom to AnnData
• Loom to SingleCellExperiment (SCE)
• SingleCellExperiment (SCE) to AnnData
• SingleCellExperiment (SCE) to Loom
• Seurat to AnnData
• Seurat to SingleCellExperiment (SCE)

Warning: Two SCEasy tools

As of the writing of this tutorial, the updated SCEasy tool is called SCEasy Converter ( Galaxy version 0.0.7+galaxy2) and it’s only available on usegalaxy.eu. The second tool is called SCEasy convert ( Galaxy version 0.0.5+galaxy1) and it works on usegalaxy.org, however has limited conversion options. Both tools should be visible on singlecell.usegalaxy.eu if you try to find them in the search box.

In this tutorial you will see multiple examples of SCEasy Tool in action. However, sometimes it is useful to know how to do this conversion manually or at least to know how it all works and better understand the structure of the files. Therefore, we also have examples showing how to convert objects manually or prepare the input files for some of our single cell tutorials and workflows.

Seurat -> AnnData

As mentioned above, there are two SCEasy tools currently available. When we speak about conversion to AnnData, the latest tool SCEasy Converter ( Galaxy version 0.0.7+galaxy2) converts Seurat into the latest version of AnnData object. However, currently it is not possible to use that output with downstream EBI single-cell tools because they only support older versions of the AnnData. If you need to use those tools, the solution is to take advantage of the second SCEasy tool SCEasy convert ( Galaxy version 0.0.5+galaxy1) which has limited conversion options but can generate AnnData files compatible with the EBI single-cell tools that we are going to use next in the workflow. Our Seurat starting file was generated by pulling data from Single Cell Expression Atlas and transforming it into the desired format, what was shown in the data import tutorial. It will be our toy dataset.

Hands-on: Get toy data

1. Create a new history for this tutorial

2. Import the AnnData object from Zenodo

   https://zenodo.org/records/10397653/files/Seurat_object.rdata
   

   Tip: Importing via links

   • Copy the link location

   • Click galaxy-upload Upload Data at the top of the tool panel

   • Select galaxy-wf-edit Paste/Fetch Data

   • Paste the link(s) into the text field

   • Press Start

   • Close the window

3. Alternatively, you can import history where we created the Seurat object: Input history

   Tip: Importing a history

   1. Open the link to the shared history
   2. Click on the Import this history button on the top left
   3. Enter a title for the new history
   4. Click on Copy History

4. Rename galaxy-pencil the dataset Seurat object

5. Check that the datatype is rdata

   Tip: Changing the datatype

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
   • In the galaxy-chart-select-data Assign Datatype, select rdata from “New type” dropdown
     • Tip: you can start typing the datatype into the field to filter the dropdown menu
   • Click the Save button

Hands-on: Choose Your Own Tutorial

This is a "Choose Your Own Tutorial" section, where you can select between multiple paths. Click one of the buttons below to select how you want to follow the tutorial

Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!

Quick one tool method Convert to AnnData object compatible with Filter, Plot, Explore workflow

AnnData -> Seurat

Let’s get an AnnData object that we can further work on. It’s the object used in many tutorials, so check it out if you’re curious.

Hands-on: Get toy data

1. Create a new history for this tutorial

2. Import the AnnData object from Zenodo

   If you do this tutorial just for learning purposes, you can download the downsampled dataset which will be much quicker to process:

   https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
   

   If you want to use the full dataset used in the other single-cell case study tutorials, here it is! Please note, it will take much longer to process it, so we will only show the conversions on the downsampled objects.

   https://zenodo.org/record/7053673/files/Mito-counted_AnnData
   

   Tip: Importing via links

   • Copy the link location

   • Click galaxy-upload Upload Data at the top of the tool panel

   • Select galaxy-wf-edit Paste/Fetch Data

   • Paste the link(s) into the text field

   • Press Start

   • Close the window

3. Rename galaxy-pencil the datasets Downsampled AnnData object

4. Check that the datatype is h5ad

   Tip: Changing the datatype

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
   • In the galaxy-chart-select-data Assign Datatype, select h5ad from “New type” dropdown
     • Tip: you can start typing the datatype into the field to filter the dropdown menu
   • Click the Save button

Hands-on: Choose Your Own Tutorial

This is a "Choose Your Own Tutorial" section, where you can select between multiple paths. Click one of the buttons below to select how you want to follow the tutorial

Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!

Quick one step method Manual conversion

AnnData -> SingleCellExperiment (SCE)

We will work on the same AnnData object so if you create a new history for this exercise, you can either get this file from Zenodo again or just copy this dataset from the previous history.

Hands-on: Get toy data

1. Create a new history for this section

2. Import the files from Zenodo

   https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
   

Tip: Copy a dataset between histories

There 3 ways to copy datasets between histories

1. From the original history

   1. Click on the galaxy-gear icon which is on the top of the list of datasets in the history panel
   2. Click on Copy Datasets

   3. Select the desired files

   4. Give a relevant name to the “New history”

   5. Validate by ‘Copy History Items’
   6. Click on the new history name in the green box that have just appear to switch to this history

2. Using the galaxy-columns Show Histories Side-by-Side

   1. Click on the galaxy-dropdown dropdown arrow top right of the history panel (History options)
   2. Click on galaxy-columns Show Histories Side-by-Side
   3. If your target history is not present
      1. Click on ‘Select histories’
      2. Click on your target history
      3. Validate by ‘Change Selected’
   4. Drag the dataset to copy from its original history
   5. Drop it in the target history

3. From the target history

   1. Click on User in the top bar
   2. Click on Datasets
   3. Search for the dataset to copy
   4. Click on its name
   5. Click on Copy to current History

Hands-on: Choose Your Own Tutorial

This is a "Choose Your Own Tutorial" section, where you can select between multiple paths. Click one of the buttons below to select how you want to follow the tutorial

Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!

Quick single tool method Convert manually

Anndata -> Cell Data Set (CDS)

Cell Data Set (CDS) format is usually used when working with a package called Monocle3 (cole-trapnell-lab). Below we show two methods on how to transform AnnData to CDS object, one of which creates an input file for Monocle 3 tutorial.

Hands-on: Choose Your Own Tutorial

This is a "Choose Your Own Tutorial" section, where you can select between multiple paths. Click one of the buttons below to select how you want to follow the tutorial

You can choose whether you want just to transform AnnData to CDS or to create CDS input file for Monocle3 tutorial to proceed with the downstream analysis described in that tutorial. Please note that depending on your dataset, you might need to refer to the first method which uses both annotated and unprocessed matrices. If you did some pre-processing on your AnnData object, then you might need to choose the first method anyway since Monocle3 performs its own pre-processing, so we would also need an unprocessed expression matrix alongside annotated (pre-processed) AnnData. That method is more expanded and specific, while the general one just shows the main principle of the conversion.

Create CDS input for Monocle3 tutorial General AnnData to CDS conversion

Downsampling FASTQ files

Sometimes, it is useful to work on smaller subsets of data (especially for teaching / learning purposes). Here is an example of how you can downsample your FASTQ files. First, let’s get some toy data. We just need two FASTQ files - one containing barcodes, the other with transcripts.

Hands-on: Get toy data

1. Create a new history for this section “Downsampling FASTQ Files”

2. Import the files from Zenodo

   https://zenodo.org/record/4574153/files/SLX-7632.TAAGGCGA.N701.s_1.r_1.fq-400k.fastq
   https://zenodo.org/record/4574153/files/SLX-7632.TAAGGCGA.N701.s_1.r_2.fq-400k.fastq
   

   Tip: Importing via links

   • Copy the link location

   • Click galaxy-upload Upload Data at the top of the tool panel

   • Select galaxy-wf-edit Paste/Fetch Data

   • Paste the link(s) into the text field

   • Press Start

   • Close the window

Funnily enough, those files are already downsampled, so you won’t have to wait for too long to download them. We are not going to analyse that data anyway, it’s just for demonstration purposes. Quickly check which file contains barcodes and which file contains transcripts. If you click on the two datasets, you will see that one has shorter sequences, while the other has longer. It’s quite straight-forward to deduce that shorter sequences are barcodes.

Hands-on: Rename the files

1. Rename file s_1.r_1 as Barcodes

2. Rename file s_1.r_2 as Transcripts

   Tip: Renaming a dataset

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, change the Name field
   • Click the Save button

Now we will convert the FASTQ files to tabular:

Hands-on: FASTQ to tabular

1. FASTQ to Tabular ( Galaxy version 1.1.5) with the following parameters:
   • param-file “FASTQ file to convert”: param-files Select multiple files: Barcodes and Transcripts
2. Rename galaxy-pencil the datasets Barcodes tabular and Transcripts tabular

Now let’s select the number of the reads we would like to keep. It’s totally up to you, we choose 100000 here.

Hands-on: Downsampling

1. Select last ( Galaxy version 1.1.0) with the following parameters:
   • param-file “Text file”: param-files Select multiple files: Barcodes tabular and Transcripts tabular
   • “Operation”: Keep last lines
   • “Number of lines”: 100000
2. Rename galaxy-pencil the dataset Barcodes cut and Transcripts cut

All done, now we just need to go back to FASTQ from Tabular again!

Hands-on: Tabular to FASTQ

1. Tabular to FASTQ ( Galaxy version 1.1.5) with the following parameters:
   • param-file “Tabular file to convert”: Barcodes cut (output of Select last tool)
   • “Identifier column”: c1 or Column 1
   • “Sequence column”: c2 or Column 2
   • “Quality column”: c3 or Column 3
2. Rename galaxy-pencil the dataset Downsampled barcode read and Downsampled transcript read

And that’s all! Your downsampled data is ready to use. You can check your answers in this example history or if you want to accelerate this process, feel free to use the workflow next time!

You've Finished the Tutorial

Please also consider filling out the Feedback Form as well!

Key points

• Single cell data from different sources may have unfamiliar formatting and thus may require different ways of ingesting it into Galaxy.

• There are many ways of importing single cell files into Galaxy and converting between single cell formats.

Frequently Asked Questions

Useful literature

Further information, including links to documentation and original publications, regarding the tools, analysis techniques and the interpretation of results described in this tutorial can be found here.

References

1. Bacon, W. A., R. S. Hamilton, Z. Yu, J. Kieckbusch, D. Hawkes et al., 2018 Single-Cell Analysis Identifies Thymic Maturation Delay in Growth-Restricted Neonatal Mice. Frontiers in Immunology 9: 10.3389/fimmu.2018.02523
2. cole-trapnell-lab monocle3. https://cole-trapnell-lab.github.io/monocle3/

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.
Did you use this material as a learner or student? Click the form below to leave feedback.

Citing this Tutorial

1. Julia Jakiela, Morgan Howells, Wendi Bacon, Converting between common single cell data formats (Galaxy Training Materials). https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-data-ingest/tutorial.html Online; accessed TODAY
2. Hiltemann, Saskia, Rasche, Helena et al., 2023 Galaxy Training: A Powerful Framework for Teaching! PLOS Computational Biology 10.1371/journal.pcbi.1010752
3. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012

BibTeX

@misc{single-cell-scrna-data-ingest,
author = "Julia Jakiela and Morgan Howells and Wendi Bacon",
	title = "Converting between common single cell data formats (Galaxy Training Materials)",
	year = "",
	month = "",
	day = ""
	url = "\url{https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-data-ingest/tutorial.html}",
	note = "[Online; accessed TODAY]"
}
@article{Hiltemann_2023,
	doi = {10.1371/journal.pcbi.1010752},
	url = {https://doi.org/10.1371%2Fjournal.pcbi.1010752},
	year = 2023,
	month = {jan},
	publisher = {Public Library of Science ({PLoS})},
	volume = {19},
	number = {1},
	pages = {e1010752},
	author = {Saskia Hiltemann and Helena Rasche and Simon Gladman and Hans-Rudolf Hotz and Delphine Larivi{\`{e}}re and Daniel Blankenberg and Pratik D. Jagtap and Thomas Wollmann and Anthony Bretaudeau and Nadia Gou{\'{e}} and Timothy J. Griffin and Coline Royaux and Yvan Le Bras and Subina Mehta and Anna Syme and Frederik Coppens and Bert Droesbeke and Nicola Soranzo and Wendi Bacon and Fotis Psomopoulos and Crist{\'{o}}bal Gallardo-Alba and John Davis and Melanie Christine Föll and Matthias Fahrner and Maria A. Doyle and Beatriz Serrano-Solano and Anne Claire Fouilloux and Peter van Heusden and Wolfgang Maier and Dave Clements and Florian Heyl and Björn Grüning and B{\'{e}}r{\'{e}}nice Batut and},
	editor = {Francis Ouellette},
	title = {Galaxy Training: A powerful framework for teaching!},
	journal = {PLoS Comput Biol} Computational Biology}
}

                   

Funding

These individuals or organisations provided funding support for the development of this resource

ELIXIR Fair Data

This work has been funded by the ELIXIR-UK FAIR Data Stewardship training UKRI award (MR/V038966/1)

Go Further

Do you want to extend your knowledge? Follow one of our recommended follow-up trainings:

• Single Cell

Galaxy Administrators: Install the missing tools

You can use Ephemeris's shed-tools install command to install the tools used in this tutorial.

shed-tools install [-g GALAXY] [-a API_KEY] -t <(curl https://training.galaxyproject.org/training-material/api/topics/single-cell/tutorials/scrna-data-ingest/tutorial.json | jq .admin_install_yaml -r)

Alternatively you can copy and paste the following YAML

---
install_tool_dependencies: true
install_repository_dependencies: true
install_resolver_dependencies: true
tools:
- name: text_processing
  owner: bgruening
  revisions: d698c222f354
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: text_processing
  owner: bgruening
  revisions: d698c222f354
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: fastq_to_tabular
  owner: devteam
  revisions: 4b347702c4aa
  tool_panel_section_label: FASTA/FASTQ
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: tabular_to_fastq
  owner: devteam
  revisions: 2dcfbbf9071a
  tool_panel_section_label: FASTA/FASTQ
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: anndata_ops
  owner: ebi-gxa
  revisions: a0274bc43b7e
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: dropletutils_read_10x
  owner: ebi-gxa
  revisions: ca62f52e0d7f
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: monocle3_create
  owner: ebi-gxa
  revisions: ada68a9d0aec
  tool_panel_section_label: Monocle3
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: scanpy_filter_cells
  owner: ebi-gxa
  revisions: 356fa6c27341
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: sceasy_convert
  owner: ebi-gxa
  revisions: f62bc418173f
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: seurat_read10x
  owner: ebi-gxa
  revisions: 301cf1b4b43d
  tool_panel_section_label: Seurat
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: regex_find_replace
  owner: galaxyp
  revisions: 399da6b5ec21
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: anndata_inspect
  owner: iuc
  revisions: 6f0d0c784f09
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: anndata_inspect
  owner: iuc
  revisions: ee98d611afc6
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: datamash_transpose
  owner: iuc
  revisions: ac092723240d
  tool_panel_section_label: Join, Subtract and Group
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: datamash_transpose
  owner: iuc
  revisions: 374cb875d38a
  tool_panel_section_label: Join, Subtract and Group
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: datamash_transpose
  owner: iuc
  revisions: a513e3fbb4c5
  tool_panel_section_label: Join, Subtract and Group
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: dropletutils
  owner: iuc
  revisions: a9caad671439
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: sceasy_convert
  owner: iuc
  revisions: 2062be7efc84
  tool_panel_section_label: Single-cell
  tool_shed_url: https://toolshed.g2.bx.psu.edu/

Feedback

No feedback has been recieved yet for this training. Be the first one by filling in the feedback form from above.