Tool Shed: sharing Galaxy tools

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Tool Shed: sharing Galaxy tools

<div class="contributors-line">
		
	
<ul class="text-list">
			
			<li>
				<a href="/training-material/hall-of-fame/abretaud/" class="contributor-badge contributor-abretaud"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/abretaud?s=36" alt="Anthony Bretaudeau avatar" width="36" class="avatar" />
    Anthony Bretaudeau</a>
			<li>
				<a href="/training-material/hall-of-fame/loraine-gueguen/" class="contributor-badge contributor-loraine-gueguen"><img src="https://avatars.githubusercontent.com/loraine-gueguen?s=36" alt="Loraine Guéguen avatar" width="36" class="avatar" />
    Loraine Guéguen</a>
			<li>
				<a href="/training-material/hall-of-fame/slegras/" class="contributor-badge contributor-slegras"><img src="https://avatars.githubusercontent.com/slegras?s=36" alt="Stéphanie Legras avatar" width="36" class="avatar" />
    Stéphanie Legras</a>
			<li>
				<a href="/training-material/hall-of-fame/bebatut/" class="contributor-badge contributor-bebatut"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/bebatut?s=36" alt="Bérénice Batut avatar" width="36" class="avatar" />
    Bérénice Batut</a></li>
</ul>

</div>

<div class="footnote" style="bottom: 8em;">
  <i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated:   
  <i class="fas fa-fingerprint" aria-hidden="true"></i><span class="visually-hidden">purl</span><abbr title="Persistent URL">PURL</abbr>: <a href="https://gxy.io/GTN:S00052">gxy.io/GTN:S00052</a>
</div>

<div class="footnote" style="bottom: 5em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> |

</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions

- What is a Tool Shed?

- How to install tools and workflows from a Tool Shed into a Galaxy instance?

- What are the Tool Shed repository types?

- How to publish with Planemo?

---

### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives

- Discover what is a Tool Shed

- Be able to install tools and workflows from a Tool Shed into a Galaxy instance

- Be able to publish tools with Planemo

---

## 1. What is a Tool Shed?

![toolshed logo with a shed over the words galaxy tool shed](../../images/toolShed.jpg)

---

### ![same logo](../../images/toolShed.jpg) = Galaxy App Store

---

### ![same logo again](../../images/toolShed.jpg) = Galaxy App Store

- Galaxy tools
- ~~Recipes for installing and compiling tools~~,
- Data Managers,
- Custom datatypes,
- Exported workflows.

---

### ![same logo again](../../images/toolShed.jpg) = Galaxy App Store

- For Galaxy administrators, facilitates:
  - **installing/updating** tools

- For tool developers, facilitates:
  - **sharing** of Galaxy utilities
  - **versioning**

---

### Available Tool Sheds

- [Galaxy community Tool Shed](https://toolshed.g2.bx.psu.edu/) -> Main Tool Shed
- [Galaxy community test Tool Shed](https://testtoolshed.g2.bx.psu.edu/) -> Sandbox for testing
- [List of public Tool Sheds](https://galaxyproject.org/toolshed/public-toolsheds/)

---

### Tool Shed interface

![toolshed interface with categories, description, and number of repositories called out with arrows](../../images/main_toolShed_ss.png)

---

### Example of tool

![view of a repository in the toolshed showing a lot of metadata](../../images/exampletool_ss.png)

---

## 2. Installing tools from a Tool Shed

---

### Connect your Galaxy to a Tool Shed

- Only for third party toolsheds (Main and Test by default)
- Add `<tool_shed />` block in `config/tool_sheds_conf.xml`
- Restart Galaxy

---

### Install a tool from the Tool Shed

- Go to the admin interface and click on "Search Tool Shed"
![admin interface of galaxy with an arrow to search tool shed](../../images/admin_searchtoolshed_ss.png)
- Select a Tool Shed

![arrow to galaxy main toolshed button in older versions of galaxy](../../images/goToToolshed_ss.png)

---

### Install a tool from the Tool Shed

- Search your tool
![toolshed search interface with arrows to the search box and one of the results, trinity](../../images/searchTool_ss.png)

---

### Install a tool from the Tool Shed

- Types of repository</span>
  - <span style="color:orange"> Tools (`$name`)</span>
  - <span style="color:blue"> Tool suites (`suite_$name`)</span>
  - <span style="color:red"> Tool dependencies (`package_$name_$version`)</span> (gradually removed, replaced by Conda dependencies)
![screenshot with multiple search results called out with boxes for the above types](../../images/typeOfRep_ss.png)

---

### Install a tool from the Tool Shed

![trinity button showing a dropdown with preview and install](../../images/goToToolInstall_ss.png)
- Install the tool
![toolshed page with a button install to galaxy](../../images/launchInstallTool_ss.png)

---

### Install a tool from the Tool Shed

- Install the tool
![another screenshot of a form with checkboxes for dependcy options](../../images/installTool_ss.png)
- <span style="font-size:70%"> Selecting Tool Shed AND conda will make Galaxy to install both</span>
- <span style="font-size:70%"> Recommended: use conda </span>

---

### Install a tool from the Tool Shed

- Check
![admin interface of galaxy with monitor installing repositories called out](../../images/monitor_installing_toolshed_rep.png)

---

### Manage installed tools

- Admin > Manage installed tools
![admin interface with manage installed tools menu item called out](../../images/manage_installed_tools.png)
![various icons used to indicate tool state](../../images/repStatus_ss.png)

---

### Manage installed tools

- Click on the name of a tool
- Manage and browse the repository
![yet another toolshed interface screenshot with the repository actions menu shown](../../images/managed_an_installed_tool.png)

---

## Tips: How to install workflows

---

###  How to install a workflow?

- Admin > Search Tool Shed
- Search the workflow
  1. ![the interface is identical to the tool install interface](../../images/toolshedWorkflow_ss.png)
  2. ![a graph with two nodes, input dataset and fastqc](../../images/schemaWorkflow_ss.png)
- <span style="font-size:70%"> Tools can come with the workflow if they are not yet installed </span>

---

### How to install a workflow?

![galaxy workflow list showing no workflows](../../images/workflowNotInstalled_ss.png)
- No workflow?

---

###  How to install a workflow?

![the installed WF is found in the manage installed tools menu](../../images/workflowInstalled_ss.png)
- Click on the imported workflow

---

###  How to install a workflow?

![contents of the repository section lists fastqc, a link to click on](../../images/makeAvailableWorkflow_ss.png)
- Click on the workflow at the bottom of the “Contents of this repository” section

---

###  How to install a workflow?

- Repository Actions > Import workflow to Galaxy

![clicking on that brings you to a view of the workflow, and you can click import to galaxy on the top right](../../images/exportWorkflow_ss.png)

---

###  How to install a workflow?

![it appears in the workflow menu](../../images/workflowOK_ss.png)
- Done!

---

## 3. Tool Shed repository types

---

### 3.1 Tool with dependencies

---

###  Tool with dependencies

![tool folder with python script, xml, and test-data is put in an archive and uploaded to the TS.](../../images/toolshedPackage.png)

---

### 3.2 Tool suite

#### Group of related repositories

---

###  Tool suite

![multiple wrappers in a single tool archive](../../images/toolSuite.png)

- Works but no way to install a single tool

---

###  Tool suite

![phylogeny pipeline containing 4 tools](../../images/toolsuite2.png)

- One repository per individual tool
- A special 'Tool suite' repository listing the individual repositories

---

###  Tool suite

- relationship between a dependent repository and one or more required repository (tools, datatypes, workflows,...)
- -> One XML file
```xml
<?xml version="1.0"?>
<repositories description="Pipeline phylogeny">
   <repository toolshed="https://testtoolshed.g2.bx.psu.edu"
     name="fasta_to_phylip" owner="gandres" changeset_revision="a895633568" />
   <repository name="mafft" owner="gandres" />
   <repository name="phyml" owner="gandres" />
[…]
</repositories>
```
No need to write it yourself, Planemo does it for you

---

###  Tool suite

![phylogeny suite gets uploaded to the TS](../../images/toolsuite3.png)

---

### 3.3 Workflow

---

###  Workflow
![the four tools from the phylogeny pipeline are connected as a workflow](../../images/workflow.png)
- A succession of tools to perform a complete analysis

---

###  Workflow

- In galaxy

![same workflow with galaxy ui](../../images/galaxyWorkflowEx.png)

---

###  Workflow

![extract workflow menu from galaxy](../../images/shareWorkflow.png)

- You get a file: Galaxy-Workflow-NAME.ga

---

###  Workflow

- You have your file Galaxy-Workflow-NAME.ga
- How to automatically install the tools used in the workflow?
  - -> `repository_dependencies.xml`
```xml
<?xml version="1.0"?>
<repositories description="Phylogeny workflow">
   <repository name="suite_phylogeny" owner="galaxy" />
</repositories>
```

---

###  Workflow

![phylogeny workflow is uploaded to the TS](../../images/workflowRepDep.png)

---

###  Workflow

![the workflow points at the tool suite which points at tools](../../images/workflowRepDep2.png)

- **Everything must be on the same toolshed**

---

## Tips

### What about tools working with *.loc files or new datatypes

---

### Tools with `*.loc` files or new datatypes

- add them to the Tool Shed repository.
- `*.loc` files and `datatypes` files should be provided as `*.sample` file in a `tool-data` directory.
- `tool-data/*.loc.sample` files come along with `tool_data_table_conf.xml.sample`.

Their content or the file themselves are put to the right location on tool set up.

---

### Tools with *.loc files or new datatypes

![toolshed interface showing some files](../../images/repWithLoc_ss.png)

<span style="font-size:60%">`tool_data_table_conf.xml.sample`</span>
```xml
<tables>
	
	<table name="bowtie2_indexes" comment_char="#">
		<columns>value, dbkey, name, path</columns>
		<file path="tool-data/bowtie2_indices.loc" />
	</table>
</tables>
```
<span style="font-size:60%">On install, the content of the file and the right
path to the *.loc file is added to `shed_tool_data_table_conf.xml`</span>

---

## 4. Publishing with Planemo

---

<i class="fas fa-exclamation-triangle" aria-hidden="true"></i><span class="visually-hidden">warning</span> *There are automated ways to publish to a ToolShed using GitHub and Continuous Integration*

[Check out these slides, "Publishing tools" section](/training-material/topics/dev/tutorials/tool-integration/slides.html)

---

### Configure the target Tool Shed

```bash
planemo config_init
```
- Creates `.planemo.yml` file in your home directory:

```yml
## Planemo Global Configuration File.
## Everything in this file is completely optional - these values can all be
## configured via command line options for the corresponding commands.
## [...]
sheds:
     toolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     testtoolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     local:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
```

---

### Configure the target Tool Shed

- Add your Tool Shed API key in `.planemo.yml`

```yml
## Planemo Global Configuration File.
## Everything in this file is completely optional - these values can all be
## configured via command line options for the corresponding commands.
## [...]
sheds:
     toolshed:
           key: "xxxxxxxxxmytoolshedAPIkeyxxxxxxx"
           #email: "<TODO>"
           #password: "<TODO>"
     testtoolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     local:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
```

---

### Configure repository

```bash
planemo shed_init --name="tandem_repeats_finder_2"
                     --owner="gandres"
                     --description="Tandem Repeats Finder description"
                     --long_description="A long long description."
                     --category="Sequence Analysis"
                     [--remote_repository_url=<URL to .shed.yml on github>]
                     [--homepage_url=<Homepage for tool.>]
```
- Creates `.shed.yml` file in the current repository (tool repository):

```yml
categories: [Sequence Analysis]
description: Tandem Repeats Finder description
long_description: A long long description.
name: tandem_repeats_finder_2
owner: gandres
```

---

### Configure repository

#### Naming conventions
<small>

| Repository Type | Recommended Name         | Examples                 |
| --------------- |--------------------------| -------------------------|
| Data Managers   | `data_manager_$name`     | `data_manager_bowtie2`   |
| Packages        | `package_$name_$version` | `package_aragorn_1_2_36` |
| Tool Suites     | `suite_$name`            | `suite_samtools`         |
| Tools           | `$name`                  | `stringle`, `bowtie2`    |

</small>
<span style="color:red">Planemo will determine the repository type based on these conventions</span>

---

### Configure repository

- Tools and tool suite `.shed.yml`: implicitly define repositories for each individual tool in the directory and build a suite for those tools.

- Example: `trinity/.shed.yml`

```yml
[...]
auto_tool_repositories:
    name_template: ""
    description_template: " (from the Trinity tool suite)"
suite:
    name: "suite_trinity"
    description: Trinity tools to assemble transcript sequences from Illumina RNA-Seq data.
```

---

### Check

```bash
planemo shed_lint --tools
```
```bash
Linting repository […]/tandem_repeats_finder
Applying linter expansion... CHECK
.. INFO: Included files all found.
Applying linter tool_dependencies_xsd... CHECK
.. INFO: tool_dependencies.xml found and appears to be valid XML
Applying linter tool_dependencies_actions... CHECK
.. INFO: Parsed tool dependencies.
Applying linter repository_dependencies... CHECK
.. INFO: No repository_dependencies.xml, skipping.
Applying linter shed_yaml... CHECK
.. INFO: .shed.yml found and appears to be valid YAML.
Applying linter readme... CHECK
.. INFO: No README found skipping.
+Linting tool […]/tandem_repeats_finder/tandem_repeats_finder_wrapper.xml
[…]
```

---

### Create repository

```bash
planemo shed_create --shed_target testtoolshed
```
- Don't forget **--shed_target**. Default is main toolshed

---

### Create repository

```bash
planemo shed_create --shed_target testtoolshed
```
```bash
Repository created
cd '[…]/tandem_repeats_finder' && git rev-parse HEAD
Repository tandem_repeats_finder_2 updated successfully.
```
![an image of a toolshed metadata from the planemo uploaded data](../../images/shedCreate.png)

---

### Update repository

```bash
planemo shed_diff --shed_target testtoolshed

```
```bash
wget -q --recursive -O - 'https://testtoolshed.g2.bx.psu.edu/repository/download?repository_id=6306166e4da9a8bf&changeset_revision=default&file_type=gz' | tar -xzf - -C /tmp/tool_shed_diff_DMcOrR/_testtoolshed_ --strip-components 1
mkdir "/tmp/tool_shed_diff_DMcOrR/_local_"; tar -xzf "/tmp/tmp5ousfg" -C "/tmp/tool_shed_diff_DMcOrR/_local_"; rm -rf /tmp/tmp5ousfg
cd "/tmp/tool_shed_diff_DMcOrR"; diff -r _local_ _testtoolshed_
diff -r _local_/tandem_repeats_finder.xml _testtoolshed_/tandem_repeats_finder.xml
41d40
<     <param name="fileout_label2" type="text" value="Unused" label="none" help="Output name for files" />
```
- The repository needs to be updated

---

### Update repository

```bash
planemo shed_update --check_diff --shed_target testtoolshed

---

### Update repository
```bash
planemo shed_update --check_diff --shed_target testtoolshed

```
![tool shed info with revision circled](../../images/checkToolUpdate.png)

- The repository has been updated

---

![planemo logo](../../images/planemo-logo.png)
#### Publishing to the Tool Shed

![planemo shed_init creates a repo, shed_lint checks, and shed_create/shed_update uploads to the TS](../../images/planemo_toolshed_publishing.png)

---

![planemo logo again](../../images/planemo-logo.png)

> Command-line utilities to assist in building and publishing Galaxy tools.

---

![and again](../../images/planemo-logo.png)

![larger graph with local development, toolshed, conda, and galaxy and various lines between them.](../../images/big_picture.png)

---

![planemo logo again](../../images/planemo-logo.png)

#### Checklist (3/3)

- Initiate the `.shed.yml` file with
```bash
planemo shed_init
```
- Create the repository on ToolShed with
```bash
planemo shed_create
```
- Install and test the wrapper on a Galaxy instance
- Improve the wrapper and push modification on toolshed
```bash
planemo shed_update
```

---

## References

- [ToolShed](https://galaxyproject.org/toolshed/tour/)
- [Install tools from the ToolShed](https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/)
- [Publish tools with Planemo](https://planemo.readthedocs.io/en/latest/)

---