Visualize EBV cube data with Panoply netCDF viewer

Overview
Creative Commons License: CC-BY Questions:
  • How to start Panoply interactive environment in Galaxy?

  • How to inspect EBV cube netCDF data?

  • How to make a plot with Panoply?

  • Where to save your plots in Panoply?

  • How to customize plots in Panoply?

  • How to generate animation in Panoply?

Objectives:
  • Learn to use Panoply in Galaxy to visuallize EBV cube netCDF geo-referenced data

  • Learn how Panoply plots are exported to Galaxy

Requirements:
Time estimation: 1 hour
Supporting Materials:
Published: Oct 28, 2022
Last modification: Jun 27, 2024
License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License. The GTN Framework is licensed under MIT
purl PURL: https://gxy.io/GTN:T00131
version Revision: 9

The practical aims at familiarzing you with the Panoply Galaxy interactive tool. Panoply is among the most popular tool to visualize geo-referenced data stored in Network Common Data Form (netCDF). It provides a graphical interface for inspecting (show metadata) and visualizing netCDF data. It supports many features to customize your plots and we will introduce some of them in this lesson.

Comment

This tutorial is significantly based on the Panoply documentation and the Galaxy climate dedicated tutorial.

In this tutorial, you will learn to:

  • Plot EBV cube arrays.
  • Use any of numerous color tables for the scale colorbar
  • Save plots to GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files.
  • Export animations as MOV or AVI videos
Agenda

In this tutorial, we will cover:

  1. NetCDF format
  2. Get data
  3. Panoply
    1. Opening up Panoply
    2. Inspect metadata
    3. Create Geo-referenced Longitude-Latitude plot
    4. Changing map projection
    5. Export Animation
    6. Quit Panoply
  4. Conclusion
Comment: Background

In this tutorial, we will be focusing on the usage of Biodiversity data in Network Common data Form (netCDF) because it is the data format used to store data on the EBV data portal. We will be using a freely available dataset representing Essential Biodiversity Variables from GEO BON data portal. We will learn to use panoply to visualize the Local bird diversity for last century.

NetCDF format

NetCDF data format is a binary format and to be able to read or visualize it, we would need to use dedicated software or libraries that can handle this “special” format. It is self-describing and machine-independent data format that supports the creation, access, and sharing of array-oriented scientific data. NetCDF files usually have the extension .nc or .netcdf.

For climate and forecast data stored in NetCDF format there are (non-mandatory) conventions on metadata (CF Convention).

In this tutorial, we will be using data from the GEO BON EBV data portal.

Get data

Hands-on: Data upload
  1. Create a new history for this tutorial. If you are not inspired, you can name it Panoply for example…

    To create a new history simply click the new-history icon at the top of the history panel:

    UI for creating new history

  2. Import the file from EBV data portal

    https://portal.geobon.org/data/upload/1/public/martins_comcom_id1_20220208_v1.nc
    
    • Copy the link location
    • Click galaxy-upload Upload Data at the top of the tool panel

    • Select galaxy-wf-edit Paste/Fetch Data
    • Paste the link(s) into the text field

    • Press Start

    • Close the window

    As an alternative to uploading the data from a URL or your computer, the files may also have been made available from a shared data library:

    1. Go into Data (top panel) then Data libraries
    2. Navigate to the correct folder as indicated by your instructor.
      • On most Galaxies tutorial data will be provided in a folder named GTN - Material –> Topic Name -> Tutorial Name.
    3. Select the desired files
    4. Click on Add to History galaxy-dropdown near the top and select as Datasets from the dropdown menu
    5. In the pop-up window, choose

      • “Select history”: the history you want to import the data to (or create a new one)
    6. Click on Import

  3. Check that the datatype is netcdf

    Files you uploaded are in netcdf format. In Galaxy, Datatypes are, by default, automatically guessed. Here, as netcdf is a derivative of the h5 format, Galaxy automatically affect the h5 datatype to netcdf files. To cope with that, one can change the datatype manually, once datasets uploaded (as shown below) OR you can directly specify datatype on the upload tool form so Galaxy will not try to automatically guess it.

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
    • In the galaxy-chart-select-data Assign Datatype, select datatypes from “New type” dropdown
      • Tip: you can start typing the datatype into the field to filter the dropdown menu
    • Click the Save button

  4. Rename Datasets galaxy-pencil

    As the original name martins_comcom_id1_20220208_v1.nc can be not so good to use, don’t hesitate to modify it in Local bird diversity (cSAR/BES-SIM) martins dataset for example.

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, change the Name field
    • Click the Save button

  5. Add a tag galaxy-tags to the dataset corresponding to #EBV

    Datasets can be tagged. This simplifies the tracking of datasets across the Galaxy interface. Tags can contain any combination of letters or numbers but cannot contain spaces.

    To tag a dataset:

    1. Click on the dataset to expand it
    2. Click on Add Tags galaxy-tags
    3. Add tag text. Tags starting with # will be automatically propagated to the outputs of tools using this dataset (see below).
    4. Press Enter
    5. Check that the tag appears below the dataset name

    Tags beginning with # are special!

    They are called Name tags. The unique feature of these tags is that they propagate: if a dataset is labelled with a name tag, all derivatives (children) of this dataset will automatically inherit this tag (see below). The figure below explains why this is so useful. Consider the following analysis (numbers in parenthesis correspond to dataset numbers in the figure below):

    1. a set of forward and reverse reads (datasets 1 and 2) is mapped against a reference using Bowtie2 generating dataset 3;
    2. dataset 3 is used to calculate read coverage using BedTools Genome Coverage separately for + and - strands. This generates two datasets (4 and 5 for plus and minus, respectively);
    3. datasets 4 and 5 are used as inputs to Macs2 broadCall datasets generating datasets 6 and 8;
    4. datasets 6 and 8 are intersected with coordinates of genes (dataset 9) using BedTools Intersect generating datasets 10 and 11.

    A history without name tags versus history with name tags

    Now consider that this analysis is done without name tags. This is shown on the left side of the figure. It is hard to trace which datasets contain “plus” data versus “minus” data. For example, does dataset 10 contain “plus” data or “minus” data? Probably “minus” but are you sure? In the case of a small history like the one shown here, it is possible to trace this manually but as the size of a history grows it will become very challenging.

    The right side of the figure shows exactly the same analysis, but using name tags. When the analysis was conducted datasets 4 and 5 were tagged with #plus and #minus, respectively. When they were used as inputs to Macs2 resulting datasets 6 and 8 automatically inherited them and so on… As a result it is straightforward to trace both branches (plus and minus) of this analysis.

    More information is in a dedicated #nametag tutorial.

Panoply

Opening up Panoply

Hands-on: Launch Panoply

Panoply is available as a Galaxy interactive tool and may not be available on all Galaxy servers.

Currently Panoply in Galaxy is available on useGalaxy.eu instance, on the “Interactive tools” tool panel section.

  1. Open the Panoply by clicking on the Panoply interactive tool shortcut
  2. Check martins_comcom_id1_20220208_v1.nc dataset selected in the netcdf input field
  3. Click Run Tool
  4. The tool will start running and will stay running permanently
  5. Click on the “User” menu at the top and go to “Active Interactive Tools” and locate the Panoply instance you started.
  6. Click on your Panoply instance
    Panoply dataset selection. Open image in new tab

    Figure 1: Select dataset
  7. Click on martins_comcom_id1_20220208_v1.nc dataset and open.

Inspect metadata

Hands-on: Inspect dataset
  1. Inspect dataset content

    Here you can look at the dataset (martins_comcom_id1_20220208_v1.nc) and related variables (crs, entity, lat, lon, metric_1, ebv_cube, time)

    Question

    what is the unit of the ebv_cube variable of metric_1and its shape?

    The unit of ebv_cube is “Percentage points”. ebv_cube is a 4D array (entity, time, latitude, longitude).

  2. Take a look at the general properties of the dataset

    Question

    Can you find the title, summary, EBV class and EBV name informations?

    Double click on “martins_comcom_id1_2022…” element of the “Name” column to display these general information Title: Local bird diversity (cSAR/BES-SIM). Summary: Changes in bird diversity at 1-degree resolution caused by land use, estimated by the cSAR model for 1900-2015 using LUH2.0 historical reconstruction of land-use. EBV Class: Community composition. EBV name: Taxonomic and phylogenetic diversity.

    Question

    Can you find the biodviersity metrics names?

    metric_1: Relative change in the number of species (%) metric_2: Absolute change in the number of species

Create Geo-referenced Longitude-Latitude plot

Hands-on: geographical map
  1. Double click on the variable ebv_cube from metric_1 and click on Create
    Panoply create Latitude longitude map. Open image in new tab

    Figure 2: Create map
    Question
    1. What does it shows?
    2. What is the date of the generated plot?
    3. Can you plot other dates?
    1. The plot represent the relative change in the number of species (%).
      Panoply Latitude longitude map. Open image in new tab

      Figure 3: Plot map
    2. The date of the default plot is 1st January 1900 at 00:00.

    3. To plot another date, change either:
      • Initial time of forecast (give a value between 1 and 12, corresponding to years between 1900 and 2010.
      • Click on the date and scroll down to select the date of your choice.
  2. Save your plot
    • Click on the tab File (from your plot window) to store your plot by selecting Save Image As
    • Double click on the folder outputs to enter this folder and save your plot. You need to make sure to save all your plot in the outputs folder otherwise you can loose all your plots once to close panoply.
  3. Change colormap Always make sure you use color blind friendly palettes.
    • To change the default colormap, click on tab “Scale” (bottom of your plot window) and select another “Color Table” (you can scroll down to go through all the different available colormap).
    • Save your plot using Save Image As and make sure to choose another name to avoid overwritting your preceding plot.
    Panoply colormap. Open image in new tab

    Figure 4: Plot colormap

Changing map projection

Hands-on: Change projection

From your previous plot window, click on Tab Map and change Projection. Try a few of them and save each of your plot with File –> Save Image As.

Panoply change projection. Open image in new tab

Figure 5: Change projection

Export Animation

Hands-on: Export animation
  1. From your previous plot window, click on File and select Export Animation. Save your plot using either MOV or AVI format. It goes through each plot e.g. for each month and create an animation where you can see the evolution of sea-ice extent from January 1979 to December 1979. You will be able to download the resulting movie from Galaxy once you quit Panoply.

Quit Panoply

Hands-on: Quit Panoply to keep your plots
  1. To make sure all your plots stored in outputs folder get exported to Galaxy, you need to quit panoply: File –> Quit Panoply.
  2. Go back to your current Galaxy history and you should find Panoply outputs
Panoply outputs. Open image in new tab

Figure 6: Panoply outputs in Galaxy

Conclusion

We have now learnt how to visualize EBV cube data using Panoply. We only use one of the two datasets so we strongly encourage you to do the same exercises with others datasets from EBV data portal.