Comment on page
Advanced Applet Tutorial
Learn to use Sambamba to create advanced Bash applets for use on the DNAnexus Platform.
In this tutorial, you'll learn to create an advanced Bash applet. Your applet will use Sambamba, an open-source toolkit, to merge multiple BAM files into a single file.
If this is your first time writing an app for use on the DNAnexus Platform, also review the Introduction to Building Apps.
When you create an applet for use on the Platform, you start by creating a local directory structure to hold your source code and other resources. For this tutorial, you'll need to create the following directory structure:
sambamba_merge_applet/
├── dxapp.json
├── Readme.md
├── resources/
│ └── usr/
│ └── bin/
│
└── src/
└── script.sh
To do this, open a terminal app on your local machine, navigate to a directory where you want to build your applet, then enter the following commands:
$ mkdir -p sambamba_merge_applet/resources/usr/bin
$ mkdir sambamba_merge_applet/src
Next use a text editor to create a file called
dxapp.json
, and, as shown in the previous section, save it in the applet's root directory, sambamba_merge_applet
. The
dxapp.json
file is a DNAnexus application metadata file. Its presence in a directory tells DNAnexus tools that it contains DNAnexus applet source code. Its component fields contain information about the applet, and specifications for how it will be run.The file's structure and content should be as follows:
{
"name": "sambamba_merge_applet",
"title": "Sambamba Mappings Merger",
"summary": "Uses Sambamba to merge multiple sorted BAM files into a single BAM file",
"version": "0.0.1",
"inputSpec":
[
{
"name": "sorted_bams",
"label": "Sorted mappings",
"help": "A set of coordinate-sorted BAM files to be merged.",
"class": "array:file",
"patterns": ["*.bam"]
},
{
"name": "advanced_options",
"label": "Advanced command line options",
"help": "Advanced command line options that will be supplied directly to the Sambamba merge execution.",
"class": "string",
"optional": true
}
],
"outputSpec": [
{
"name": "merged_bam",
"label": "Merged sorted mappings",
"help": "A BAM file with the merged mappings.",
"class": "file",
"patterns": ["*.bam"]
}
],
"runSpec": {
"interpreter": "bash" ,
"file": "src/script.sh",
"systemRequirements": {
"*": {
"instanceType": "mem2_ssd1_v2_x4"
}
},
"distribution": "Ubuntu",
"release": "20.04",
"execDepends": []
},
"openSource": true
}
The applet's name -
sambamba_merge_applet
- is specified in the name
field. The
inputSpec
field is an array containing two objects. Each provides specs for one of the two inputs taken by the applet:- 1.
sorted_bams
- This input consists of an array of BAM files. As indicated in thepatterns
subfield, each of these files must have a name that ends with the extension.bam
. - 2.
advanced_options
- This optional input consists of a string of advanced command-line options to be passed to Sambamba, for use in merging the source BAM files. See the Sambamba documentation for more on these options. See below for details on how to include these options when your applet launches Sambamba.
The
outputSpec
field is also an array, containing a single object that provides specs for the applet's single output:- 1.
merged_bam
- A single BAM file, which, as indicated in thepatterns
subfield, must have a name that ends with the extension.bam
.
The value "bash" in the
runSpec
field's interpreter
subfield specifies that the applet is a Bash script. The value "src/script.sh" in the
runSpec
field's file
subfield specifies that the worker running the applet should run the executable script.sh
, located in the applet's src
subdirectory.In the
runSpec
field'ssystemRequirements
subfield, note the value "mem2_ssd1_v2_x4" in the instanceType
field. This specifies that each of the applet's entry points should be run using the mem2_ssd1_v2_x4 instance type.Download the Sambamba binary from the Sambamba releases page. Uncompress the executable and and place it in your applet's
resources/usr/bin/
subdirectory.Now run the following commands:
# Navigate to your applet root directory
cd /path/to/app/directory
# Untar the downloaded executable
tar -xzf /path/to/downloaded/sambamba_executable
# Rename and move the executable to the correct directory
# Note: if you don't rename the executable, make sure the
# app source code uses the full name of the downloaded
# sambamba executable.
mv sambamba_* resources/usr/bin/sambamba
The next step is to write the script that will be executed when the applet is launched.
Start by using a text editor to create a file named
script.sh
and save it in the applet's src
subdirectory.Add these two lines at the top of the file:
#!/bin/bash
set -e -x -o pipefail
The first line, sometimes called the "shebang," specifies which interpreter should be used to parse the remainder of the file - in this case, the Bash interpreter.
The second line contains settings to be used in executing the script:
- The
-e
flag ensures that execution will abort on an error. - The
-o pipefail
ensures that Bash will throw an error if it encounters an error within a pipeline. - The
-x
flag ensures that Bash will output each line as it is executed. This is useful for debugging.
Note that on the DNAnexus Platform, workers have the flag
-e
set by default. If you would like to keep your script to the end, regardless of any errors encountered during execution, replace -e
with +e
in the second line of the script.On the DNAnexus Platform, the execution of an applet on a worker runs in the directory defined by the constant
$HOME
. To be accessible to the execution, inputs need to be in the subdirectory $HOME/in
. Add the following line to your script file, to have your input files automatically downloaded to $HOME/in
:$ dx-download-all-inputs
Note that you named your applet's first input
sorted_bams
, and defined it as an array of files. As such, when your input files are downloaded to $HOME/in
, they will be placed in a subdirectory $HOME/in/sorted_bams
. Each file will be placed in a separate subdirectory within $HOME/in/sorted_bams
, with these subdirectories named with integers starting with 0, like the elements of an array.For example, if you supply three files to the applet, named
SRR100022_chrom20_mapped_to_b37.bam
, SRR100022_chrom21_mapped_to_b37.bam
, and SRR100022_chrom22_mapped_to_b37.bam
, they will be downloaded to:$HOME/in/sorted_bams/0/SRR100022_chrom20_mapped_to_b37.bam
$HOME/in/sorted_bams/1/SRR100022_chrom21_mapped_to_b37.bam
$HOME/in/sorted_bams/2/SRR100022_chrom22_mapped_to_b37.bam
The file structure for the inputs will then be as follows:
$HOME
├── in
│ └── sorted_bams
│ ├── 0
│ │ └── SRR100022_chrom20_mapped_to_b37.bam
│ ├── 1
│ │ └── SRR100022_chrom21_mapped_to_b37.bam
│ └── 2
│ └── SRR100022_chrom22_mapped_to_b37.bam
│ ...
Next add the following line to your script, creating a directory
$HOME/out/merged_bam
, to store your output file:$ mkdir -p out/merged_bam
Note that the directory's name,
merged_bam
, corresponds to the name of your applet's output parameter, as specific in the dxapp.json
file. This ensures that your applet will treat files in this directory as outputs, and then automatically upload them after execution has finished.On the DNAnexus Platform, environment variables can be used to set execution output filenames to follow a specific pattern. To leverage this feature, add the following line to your script:
$ output_name="${sorted_bams_prefix[0]}_merged.bam"
This line specifies that your output file will have a name with the following components:
- 1.
${sorted_bams_prefix[0]}
- This adds to the output filename the prefix of the first file used as an input, i.e. that file's name, with its extension or extensions removed.sorted_bams
refers to the name of the array that contains your input files._prefix
and[0]
specify the prefix of the file stored in the first element of that array. Wrapping this all in${}
ensures that it is parsed as code rather than as a string. - 2.
_merged
- This string adds an additional descriptive element to the filename, making clear it consists of multiple files merged into one. - 3.
.bam
- This extension defines the file as a.bam
, or Binary Alignment Map, file.
Next add a line to your script that will launch Sambamba and have it merge your input files into a single file:
sambamba merge $advanced_options "$output_name" "${sorted_bams_path[@]}"
To break this down:
sambamba merge
invokes Sambamba's merge function.$advanced_options
are any advanced Sambamba command-line options you specified, when creating the applet's dxapp.json file."$output_name"
gives the output file the name that, in the previous step, you stored in the variableoutput_name
."${sorted_bams_path[@]}"
specifies that Sambamba should use as inputs the files stored in the arraysorted_bams
. Adding_path
enables your script to leverage a DNAnexus Platform environment variable to specify that Sambamba should look for this array in the directory$HOME/in/sorted_bams/
To ensure that
${sorted_bams_path[@]}
enables your script to find your input files and provide them to Sambamba, you must use dx-download-all-inputs
to download those files to $HOME/in
, as per the instructions above.When your script is run, Bash will automatically interpret the variables you've included in the
merge
command. So if, for example, you have three input files named NA12878.chr1.bam
, NA12878.chr2.bam
, and NA12878.chr3.bam
Bash will interpret your code as follows:sambamba merge $HOME/out/bam_output/part_0_merged.bam $HOME/in/sorted_bams/0/NA12878.chr1.bam $HOME/in/sorted_bams/1/NA12878.chr2.bam $HOME/in/sorted_bams/2/NA12878.chr3.bam -out
You can use the following merge command in your shell script, if for some reason you don't want it to leverage DNAnexus Platform environment variables:
sambamba merge $advanced_options $output_folder/$output_name $HOME/in/sorted_bams/*/*
After Sambamba merges your input files, the output file needs to be moved to the
$HOME/out/merged_bam
folder on the worker. To provide for this, add the following line to your script:$ mv $output_name out/merged_bam/
Then your output file needs to be uploaded from the worker to the DNAnexus Platform. For this, use the utility
dx-upload-all-outputs
, which will automatically upload the contents of all subdirectories on the path $HOME/out/
. To provide for this, add the following line to your script:$ dx-upload-all-outputs
You've completed your script. It should read as follows:
#!/bin/bash
set -e -x -o pipefail
dx-download-all-inputs
mkdir -p out/merged_bam
output_name="${sorted_bams_prefix[0]}_merged.bam"
sambamba merge $advanced_options "$output_name" "${sorted_bams_path[@]}"
mv "$output_name" out/merged_bam/
dx-upload-all-outputs
You're ready to build and run your applet.
If you haven't yet done so, log into the DNAnexus Platform using your terminal app. Then select the project in which you'd like to work.
If you have BAM files you'd like to merge using your applet, upload them to your project. If you prefer, you can use the BAM files available in the Demo Data public project.
If you choose to upload your own data, test your applet by doing an initial run that uses small files.
Now use dx build to build your applet. Enter the following command
In your terminal app, enter the following commands to build and run your applet:
$ dx build path/to/app/directory
$ dx run sambamba_merge_applet
In this tutorial you manually created the applet local directory,
dxapp.json
, and shell script (src/script.sh
). These steps can be automated, by using dx-app-wizard
. Consult the Intro to Building Apps tutorial for guidance on using dx-app-wizard
. Note that
dx-app-wizard
does have certain limitations. It does not, for example, prompt you to provide advanced configuration settings, such as instanceType
specs, and the patterns
settings you added here to your applet's inputSpec
and outputSpec
definitions. In addition, it does not use either dx-download-all-inputs
or dx-upload-all-outputs
.dx-app-wizard
can still be useful as a tool to speed your work, even if you want to leverage the advanced functionality available when you manually perform all the steps involved in creating an applet. You can, for example, use dx-app-wizard
to create the applet's directory structure and a basic dxapp.json
file. Then use a text editor to, as needed, add additional fields to the dxapp.json
file, and replace the Bash script created by dx-app-wizard
, with your own script.See the Developer Tutorials page for language-specific tutorials. Each provides guidance in crafting complex applets and apps, in a particular language.
You can download the source code of a number of open-source apps available for use on the DNAnexus Platform, such as Cloud Workstation.
To get a list of these apps, use the following command:
$ dx api system findApps '{"describe":{"fields":{"openSource": true, "name": true}}}'| \
jq '.results|.[]|select(.describe.openSource)|.describe.name'
To download the source code of an open-source app available on the Platform, use
dx get
as follows, swapping in the app's name for app-cloud_workstation
:$ dx get app-cloud_workstation
Last modified 6mo ago