Bash Apps
Learn to write a basic Bash app .
If you have not already, you should install the DNAnexus SDK and walk through the Intro to Building Apps tutorial.
Bash apps are the simplest apps you can create on DNAnexus. They include a shell script which runs on a virtual machine on the cloud. The script, written by the app developer, handles downloading inputs, processing, and uploading outputs.
Downloading and Using File Inputs
Two methods are available to download inputs: one by one, or all at once.
To download all inputs at once (recommended), use the dx-download-all-inputs
utility. Add --parallel
to allow multiple downloads in parallel: dx-download-all-inputs --parallel
.
Inputs are downloaded to a folder called "in" under the home folder. Each input is placed under its own subfolder ~/in/name_of_input_field/
, named after the input field. Files keep their original filenames as supplied by the user who launched the applet.
For example, if your applet defines a file input called mappings
and a user runs it with a file called SRR001.bam
, it will be downloaded into ~/in/mappings/SRR001.bam
.
The system defines the following helper bash variables, which you can use in your applet:
$mappings_path
~/in/mappings/SRR001.bam
$mappings_name
SRR001.bam
$mappings_prefix
SRR001
$mappings
{"$dnanexus_link": "file-F77Bp7002302Zb343BF1FpG0"}
To download inputs one by one, use the following syntax:
# Download the input file to the current working directory
dx download "$name_of_input_field"
This will download the file to the current working directory, retaining the original filename as supplied by the user who launched the applet. You can use the "$xxxxxx_name"
variable (as shown in the table above) to refer to that filename.
To name the local file using a different name than the original filename, use the following syntax:
# Download the input file and save it as 'local_filename'
dx download "$name_of_input_field" -o local_filename
To stream the local file and pipe it to another command, use the following syntax:
# Stream the input file and pipe it to another command
dx cat "$name_of_input_field" | command
You can combine these strategies. For example, you can download or stream some inputs one by one, and the rest all at once, using the following syntax:
# Download all inputs except the specified ones
dx-download-all-inputs --except name_of_input_field1 --except name_of_input_field2
# Download a specific input file
dx download "$name_of_input_field1"
# Stream another input file and pipe it to a command
dx cat "$name_of_input_field2" | command
Downloading Inputs Using dx-mount-all-inputs
An alternative to dx-download-all-inputs
is the dx-mount-all-inputs
command-line utility. You can use it by adding this line to your script.sh
:
dx-mount-all-inputs
dx-mount-all-inputs
uses the same directory structure as illustrated above for dx-download-all-inputs
, except that the files are mounted to the respective location rather than downloaded. When using dx-mount-all-inputs
, input files do not take up local storage because they are mounted using Linux FUSE technology and streamed behind the scenes transparently when accessed.
Uploading Outputs
Two methods are available to upload outputs: one by one, or all at once.
To upload outputs one by one, use the following syntax:
# Upload a file and capture its file ID
id=$(dx upload /path/to/local/file --brief)
# Register the uploaded file as an output
dx-jobutil-add-output name_of_output_field "$id"
If you would like the uploaded file to have a different name than the local file, add the following to the dx upload
command:
# Upload a file with a different remote filename
dx upload /path/to/local/file --brief --path remote_filename
If you would like the uploaded file to appear under a subfolder in the applet outputs, add the following to the dx upload
command:
# Upload a file to a subfolder in the output
dx upload /path/to/local/file --brief --path /subfolder/remote_filename --parents
If you would like the uploaded file to contain metadata, for example, set a property (key/value pair), add the following to the dx upload
command:
# Upload a file with a property (metadata)
dx upload /path/to/local/file --brief --property key=value
To upload all outputs at once, create a folder "out" under the home folder, and create a subfolder named after each output field. Place a file under each subfolder, and call the dx-upload-all-outputs
utility:
# Create output subfolders for each output field
mkdir -p ~/out/name_of_output_field1/ ~/out/name_of_output_field2/
# Move files into their respective output subfolders
mv file1 ~/out/name_of_output_field1/
mv file2 ~/out/name_of_output_field2/
# Upload all outputs at once
dx-upload-all-outputs
If you would like any uploaded file to have a different name than the local file, rename them as you move them:
# Rename file as you move it to the output subfolder
mv file1 ~/out/name_of_output_field1/renamed_file1
If you would like any uploaded file to appear under a subfolder in the applet outputs, create the subfolder like this:
# Create a subfolder and move the file into it
mkdir -p ~/out/name_of_output_field1/subfolder/
mv file1 ~/out/name_of_output_field1/subfolder/
If you would like the uploaded files to contain metadata, for example, set a property (key/value pair), make the following changes:
Install the
attr
Linux executable by specifying theattr
Ubuntu package in yourdxapp.json
.For each file, and for each key/value pair you would like to attach, set the respective Linux extended file system attribute like this:
# Set an extended attribute (property) on a file attr -s key -V value ~/out/name_of_output_field1/file1
Add
--xattr-properties
to thedx-upload-all-outputs
invocation:# Upload all outputs and include extended attributes as properties dx-upload-all-outputs --xattr-properties
Last updated
Was this helpful?