Advanced Applet Tutorial

Learn to use Sambamba to create advanced Bash applets for use on the DNAnexus Platform.

In this tutorial, you'll learn to create an advanced Bash applet. Your applet will use Sambamba, an open-source toolkit, to merge multiple BAM files into a single file.

Before You Begin

Learning about Sambamba

If you're not familiar with Sambamba, review the Sambamba documentation.

Working on the DNAnexus Platform

Download the DNAnexus SDK if you haven't already done so.

If you're not familiar with the dx command-line client, review the Command-Line Quickstart.

If this is your first time writing an app for use on the DNAnexus Platform, also review the Introduction to Building Apps.

Step 1. Create the Applet Directory and Subdirectories

When you create an applet for use on the Platform, you start by creating a local directory structure to hold your source code and other resources. For this tutorial, you'll need to create the following directory structure:

sambamba_merge_applet/
├── dxapp.json
├── Readme.md
├── resources/
│   └── usr/
│       └── bin/
│
└── src/
    └── script.sh

To do this, open a terminal app on your local machine, navigate to a directory where you want to build your applet, then enter the following commands:

$ mkdir -p sambamba_merge_applet/resources/usr/bin
$ mkdir sambamba_merge_applet/src

Step 2. Create the dxapp.json File

Next use a text editor to create a file called dxapp.json, and, as shown in the previous section, save it in the applet's root directory, sambamba_merge_applet.

The dxapp.json file is a DNAnexus application metadata file. Its presence in a directory tells DNAnexus tools that it contains DNAnexus applet source code. Its component fields contain information about the applet, and specifications for how it will be run.

The file's structure and content should be as follows:

{
  "name": "sambamba_merge_applet",
  "title": "Sambamba Mappings Merger",
  "summary": "Uses Sambamba to merge multiple sorted BAM files into a single BAM file",
  "version": "0.0.1",
  "inputSpec":
  [
    {
      "name": "sorted_bams",
      "label": "Sorted mappings",
      "help": "A set of coordinate-sorted BAM files to be merged.",
      "class": "array:file",
      "patterns": ["*.bam"]
    },

    {
      "name": "advanced_options",
      "label": "Advanced command line options",
      "help": "Advanced command line options that will be supplied directly to the Sambamba merge execution.",
      "class": "string",
      "optional": true
    }
  ],
  "outputSpec": [
    {
       "name": "merged_bam",
      "label": "Merged sorted mappings",
      "help": "A BAM file with the merged mappings.",
      "class": "file",
       "patterns": ["*.bam"]
    }
  ],
  "runSpec": {
    "interpreter": "bash" ,
    "file": "src/script.sh",
    "systemRequirements": {
      "*": {
        "instanceType": "mem2_ssd1_v2_x4"
      }
    },    
    "distribution": "Ubuntu",
    "release": "20.04",
    "execDepends": []
  },
  "openSource": true
}

Applet Name

The applet's name - sambamba_merge_applet - is specified in the name field.

Inputs and Outputs

The inputSpec field is an array containing two objects. Each provides specs for one of the two inputs taken by the applet:

1. sorted_bams- This input consists of an array of BAM files. As indicated in the patterns subfield, each of these files must have a name that ends with the extension .bam.

2. advanced_options - This optional input consists of a string of advanced command-line options to be passed to Sambamba, for use in merging the source BAM files. See the Sambamba documentation for more on these options. See below for details on how to include these options when your applet launches Sambamba.

The outputSpec field is also an array, containing a single object that provides specs for the applet's single output:

1. merged_bam- A single BAM file, which, as indicated in the patterns subfield, must have a name that ends with the extension .bam.

Run Specs

The value "bash" in the runSpec field's interpreter subfield specifies that the applet is a Bash script.

The value "src/script.sh" in the runSpec field's file subfield specifies that the worker running the applet should run the executable script.sh, located in the applet's src subdirectory.

In the runSpec field'ssystemRequirements subfield, note the value "mem2_ssd1_v2_x4" in the instanceType field. This specifies that each of the applet's entry points should be run using the mem2_ssd1_v2_x4 instance type.

Step 3. Package the Executable With the Applet

Download the Sambamba binary from the Sambamba releases page. Uncompress the executable and and place it in your applet's resources/usr/bin/ subdirectory.

Now run the following commands:

# Navigate to your applet root directory
cd /path/to/app/directory

# Untar the downloaded executable
tar -xzf /path/to/downloaded/sambamba_executable

# Rename and move the executable to the correct directory
# Note: if you don't rename the executable, make sure the
#       app source code uses the full name of the downloaded
#       sambamba executable.
mv sambamba_* resources/usr/bin/sambamba

Step 4. Write the Applet Script

The next step is to write the script that will be executed when the applet is launched.

Script Setup

Start by using a text editor to create a file named script.sh and save it in the applet's src subdirectory.

Add these two lines at the top of the file:

#!/bin/bash
set -e -x -o pipefail

The first line, sometimes called the "shebang," specifies which interpreter should be used to parse the remainder of the file - in this case, the Bash interpreter.

The second line contains settings to be used in executing the script:

• The -e flag ensures that execution will abort on an error.

• The -o pipefail ensures that Bash will throw an error if it encounters an error within a pipeline.

• The -x flag ensures that Bash will output each line as it is executed. This is useful for debugging.

Note that on the DNAnexus Platform, workers have the flag -e set by default. If you would like to keep your script to the end, regardless of any errors encountered during execution, replace -e with +e in the second line of the script.

Configuring Inputs

On the DNAnexus Platform, the execution of an applet on a worker runs in the directory defined by the constant $HOME. To be accessible to the execution, inputs need to be in the subdirectory $HOME/in. Add the following line to your script file, to have your input files automatically downloaded to $HOME/in:

$ dx-download-all-inputs

Note that you named your applet's first input sorted_bams, and defined it as an array of files. As such, when your input files are downloaded to $HOME/in, they will be placed in a subdirectory $HOME/in/sorted_bams. Each file will be placed in a separate subdirectory within $HOME/in/sorted_bams, with these subdirectories named with integers starting with 0, like the elements of an array.

For example, if you supply three files to the applet, named SRR100022_chrom20_mapped_to_b37.bam, SRR100022_chrom21_mapped_to_b37.bam, and SRR100022_chrom22_mapped_to_b37.bam, they will be downloaded to:

$HOME/in/sorted_bams/0/SRR100022_chrom20_mapped_to_b37.bam $HOME/in/sorted_bams/1/SRR100022_chrom21_mapped_to_b37.bam $HOME/in/sorted_bams/2/SRR100022_chrom22_mapped_to_b37.bam

The file structure for the inputs will then be as follows:

$HOME
├── in
│   └── sorted_bams
│       ├── 0
│       │   └── SRR100022_chrom20_mapped_to_b37.bam
│       ├── 1
│       │   └── SRR100022_chrom21_mapped_to_b37.bam
│       └── 2
│           └── SRR100022_chrom22_mapped_to_b37.bam
│ ...

Configuring Outputs

Creating an Output Directory

Next add the following line to your script, creating a directory $HOME/out/merged_bam, to store your output file:

$ mkdir -p out/merged_bam

Note that the directory's name, merged_bam, corresponds to the name of your applet's output parameter, as specific in the dxapp.json file. This ensures that your applet will treat files in this directory as outputs, and then automatically upload them after execution has finished.

Setting the Output Filename

On the DNAnexus Platform, environment variables can be used to set execution output filenames to follow a specific pattern. To leverage this feature, add the following line to your script:

$ output_name="${sorted_bams_prefix[0]}_merged.bam"

This line specifies that your output file will have a name with the following components:

1. ${sorted_bams_prefix[0]} - This adds to the output filename the prefix of the first file used as an input, i.e. that file's name, with its extension or extensions removed. sorted_bams refers to the name of the array that contains your input files. _prefix and [0] specify the prefix of the file stored in the first element of that array. Wrapping this all in ${} ensures that it is parsed as code rather than as a string.

2. _merged - This string adds an additional descriptive element to the filename, making clear it consists of multiple files merged into one.

3. .bam - This extension defines the file as a .bam, or Binary Alignment Map, file.

Setting Up the File Merge

Next add a line to your script that will launch Sambamba and have it merge your input files into a single file:

sambamba merge $advanced_options "$output_name" "${sorted_bams_path[@]}"

To break this down:

• sambamba merge invokes Sambamba's merge function.

• $advanced_options are any advanced Sambamba command-line options you specified, when creating the applet's dxapp.json file.

• "$output_name" gives the output file the name that, in the previous step, you stored in the variable output_name.

• "${sorted_bams_path[@]}" specifies that Sambamba should use as inputs the files stored in the array sorted_bams. Adding _path enables your script to leverage a DNAnexus Platform environment variable to specify that Sambamba should look for this array in the directory $HOME/in/sorted_bams/

To ensure that ${sorted_bams_path[@]} enables your script to find your input files and provide them to Sambamba, you must use dx-download-all-inputs to download those files to $HOME/in, as per the instructions above.

When your script is run, Bash will automatically interpret the variables you've included in the merge command. So if, for example, you have three input files named NA12878.chr1.bam, NA12878.chr2.bam, and NA12878.chr3.bam Bash will interpret your code as follows:

sambamba merge $HOME/out/bam_output/part_0_merged.bam $HOME/in/sorted_bams/0/NA12878.chr1.bam $HOME/in/sorted_bams/1/NA12878.chr2.bam $HOME/in/sorted_bams/2/NA12878.chr3.bam -out

Setting Up the File Merge Without Using Environment Variables

You can use the following merge command in your shell script, if for some reason you don't want it to leverage DNAnexus Platform environment variables:

sambamba merge $advanced_options $output_folder/$output_name $HOME/in/sorted_bams/*/*

Uploading Results with dx-upload-all-outputs

After Sambamba merges your input files, the output file needs to be moved to the $HOME/out/merged_bam folder on the worker. To provide for this, add the following line to your script:

$ mv $output_name out/merged_bam/

Then your output file needs to be uploaded from the worker to the DNAnexus Platform. For this, use the utility dx-upload-all-outputs , which will automatically upload the contents of all subdirectories on the path $HOME/out/. To provide for this, add the following line to your script:

$ dx-upload-all-outputs

Step 5. Build and Run the Applet

You've completed your script. It should read as follows:

#!/bin/bash
set -e -x -o pipefail
dx-download-all-inputs
mkdir -p out/merged_bam
output_name="${sorted_bams_prefix[0]}_merged.bam"
sambamba merge $advanced_options "$output_name" "${sorted_bams_path[@]}"
mv "$output_name" out/merged_bam/
dx-upload-all-outputs

You're ready to build and run your applet.

If you haven't yet done so, log into the DNAnexus Platform using your terminal app. Then select the project in which you'd like to work.

Selecting BAM Files to Merge

If you have BAM files you'd like to merge using your applet, upload them to your project. If you prefer, you can use the BAM files available in the Demo Data public project.

If you choose to upload your own data, test your applet by doing an initial run that uses small files.

Building and Running Your Applet

Now use dx build to build your applet. Enter the following command

In your terminal app, enter the following commands to build and run your applet:

$ dx build path/to/app/directory
$ dx run sambamba_merge_applet

Learn More

Using dx-app-wizard

In this tutorial you manually created the applet local directory, dxapp.json, and shell script (src/script.sh). These steps can be automated, by using dx-app-wizard. Consult the Intro to Building Apps tutorial for guidance on using dx-app-wizard.

Note that dx-app-wizard does have certain limitations. It does not, for example, prompt you to provide advanced configuration settings, such as instanceType specs, and the patterns settings you added here to your applet's inputSpec and outputSpec definitions. In addition, it does not use either dx-download-all-inputs or dx-upload-all-outputs.

dx-app-wizard can still be useful as a tool to speed your work, even if you want to leverage the advanced functionality available when you manually perform all the steps involved in creating an applet. You can, for example, use dx-app-wizard to create the applet's directory structure and a basic dxapp.json file. Then use a text editor to, as needed, add additional fields to the dxapp.json file, and replace the Bash script created by dx-app-wizard, with your own script.

Language-Specific Tutorials

See the Developer Tutorials page for language-specific tutorials. Each provides guidance in crafting complex applets and apps, in a particular language.

Getting Sample Code

You can download the source code of a number of open-source apps available for use on the DNAnexus Platform, such as Cloud Workstation.

To get a list of these apps, use the following command:

$ dx api system findApps '{"describe":{"fields":{"openSource": true, "name": true}}}'| \
  jq '.results|.[]|select(.describe.openSource)|.describe.name'

To download the source code of an open-source app available on the Platform, use dx get as follows, swapping in the app's name for app-cloud_workstation:

$ dx get app-cloud_workstation