Developing Apps and Applets

What's the difference between an app and an applet?
Applets and apps are both executables in the platform that you can run in the cloud. You can mix and match them in the same workflow. They can also both be specified to require special permissions to the project context or to the user's other projects. Here's how they're different:
​
Applets
Apps
Model
can be used as scripts for manipulating data, creating proprietary analysis pipelines, or testing versions before publishing an app; they are easy to create and revise
are general-purpose tools of interest to the community at large that usually strive for compatibility, reproducibility, and robustness
Platform Representation
are data objects residing in projects
are created from applets and reside in separate data containers outside of users' projects; there is one container per version of an app
Input/Output Specifications
can be created with no input or output specifications; this means that it can be run with any input, and it makes no guarantees about what it returns
must have input and output specifications so that they behave predictably and can be used compatibly with other apps
Sharing
are shared by sharing the project in which they live, and are otherwise completely private
are shared by publishing the app to a customizable list of authorized users that can easily discover (via the website or command-line), install, and run it
Open/Closed Source
expose their source code and attached resources to anyone who has VIEW permissions to their project
can hide their source code and attached resources so that only the app's developers can access them
Naming
can be given any name
belong to a global namespace, and the first time someone creates an app with a particular name, it is reserved for that user and any others that they designate as developers
Versioning
each revision has a permanent unique ID, recorded in any job started and any data produced
are published with semantic version numbers (xx.yy.zz), with different versions automatically archived and accessible to platform users
Want to read more? Check out the API documentation for applets and apps.
So should I be building an applet or an app?
If you are new to writing applications for the DNAnexus platform, start by building applets. They are easier to iterate quickly and share directly with collaborators in your project. Keep in mind that as long as you write your applet to have both an input and output specification (which we recommend as a good coding practice in general), it can always be built to become an app later.
How do I package a Linux executable into an app(let)?
If you haven't already, you may want to take a look at the Intro to Building Apps tutorial, which walks you through creating an applet that takes in a file and outputs another file.
A brief overview of the steps involved:
1.Run dx-app-wizard:
Pick some app name (e.g. appname)
Specify any input files or other parameters needed for the executable
Specify any output files or values that the executable generates
Pick bash as the language
Pick the basic template
2.Place the executable you want to run into the appname/resources directory.
3.Add a line in the .sh file in the appname/src directory to run the executable on the file with any parameters received from the input. (Use the lines generated by the app wizard for automatically downloading any file input and uploading file output.)
4.Run either dx build or dx build --create-app (depending on whether you want to build an applet or an app).
How do I install software requirements for my app (e.g. Java, R, samtools)?
There are a few different options, depending on where the software can be found and if you would like it to be reproducible.
1.If the software you need is available as an APT package in Ubuntu , you can edit the dxapp.json file to specify it (and, optionally, the version you want). The following JSON excerpt shows you how to request APT software packages; in this case, Java, R, and the samtools packages have been requested and will be available when the app is run.
1
{ "runSpec": {
2
"execDepends": [
3
    {"name": "openjdk-6-jre", "version": "6b24-1.11.1-4ubuntu2"},
4
    {"name": "r-base-core"},
5
    {"name": "samtools"},
Copied!
Note that the APT "Recommends" of the packages are not installed by default. You can simulate this behavior on your local Ubuntu machine with --no-install-recommends option to apt-get install.
2.If the software resides in a globally accessible location (e.g. a git repository hosted on GitHub), you must also request network access to the server hosting it. The following JSON excerpt shows how to request and automatically build a repository hosted on GitHub. (Alternatively, you can also just request access to a particular host and perform the download and build steps manually as part of your app.)
1
{ "access": {
2
"network": ["github.com"]
3
},
4
"runSpec": {
5
"execDepends": [
6
    {"name": "dx-toolkit",
7
     "package_manager": "git",
8
     "url": "[email protected]:dnanexus/dx-toolkit.git",
9
     "tag": "master",
10
     "build_commands": "make install DESTDIR=/ PREFIX=/opt/dnanexus"
11
    },
Copied!
3.If it is software that you already have and would like to upload and install as part of the app, then you should place any necessary files in the resources directory of your app before running dx build. The build tool will compress and package up the contents of that directory as part of your app, and, when it is run, it will be automatically downloaded and extracted into the root directory /. Your first steps in the code of your app should then be to perform any build or installation commands necessary.
4.You can build an Asset Bundle, which is compatible with software from both APT and GitHub. It can also be compatible with other software. Please see the Asset Build Process page for more details.
NOTE: The following Java 8 packages are not available via traditional apt-get; however, we have manually injected them into our APT repo. As a result, you can install them using method #1: the runSpec field of the app's dxapp.json file.
openjdk-8-dbg
openjdk-8-demo
openjdk-8-doc
openjdk-8-jdk
openjdk-8-jre
openjdk-8-jre-headless
openjdk-8-jre-jamvm
openjdk-8-jre-zero
openjdk-8-source
How do I write my app in my favorite programming language?
The following languages are fully supported via dx-app-wizard templates, client libraries and tools, and sample code:
​Python​
​bash​
If you would like to use a language that is not yet supported, you can still do so by packaging any scripts or files with a Python or bash script that will be responsible for running your code. See the Intro to Building Apps tutorial for an example of how to package up an arbitrary Linux executable as an applet. If you need to install any extra software dependencies such as Java or R, see [the above answer](/FAQ#How-do-I-install-software-requirements-for-my-app-(e.g.-Java,-R,-samtools)%3F) for more information.
How do I request more memory/CPU for my app? How do I specify the compute instance type?
By default, a job will run on a virtual machine with these resources:
Dual-core x86-64 CPU
7.5 GB of RAM
400 GB scratch file system
To request a different machine, you will need to edit the dxapp.json file to specify the instance type for each entry point of your app. Please see the documentation on the Run Specification for the list of available instance types and other details.
The following dxapp.json excerpt shows how to request larger virtual machines for both the main and myEntryPoint entry points of your app; any other entry points not listed in systemRequirements will use the default virtual machine mem2_hdd2_x2.
1
{ "runSpec": {
2
    "systemRequirements": {
3
      "main": {
4
        "instanceType": "mem2_hdd2_x4"
5
      },
6
      "myEntryPoint": {
7
        "instanceType": "mem3_hdd2_x2"
8
      }
9
    },
Copied!
What are the default user limits for processes running inside the Linux execution environment?
The default limits, as presented by the output of ulimit -a, are the following:
1
core file size          (blocks, -c) 0
2
data seg size           (kbytes, -d) unlimited
3
scheduling priority             (-e) 0
4
file size               (blocks, -f) unlimited
5
pending signals                 (-i) 59461
6
max locked memory       (kbytes, -l) 64
7
max memory size         (kbytes, -m) unlimited
8
open files                      (-n) 4096
9
pipe size            (512 bytes, -p) 8
10
POSIX message queues     (bytes, -q) 819200
11
real-time priority              (-r) 0
12
stack size              (kbytes, -s) 8192
13
cpu time               (seconds, -t) unlimited
14
max user processes              (-u) 59461
15
virtual memory          (kbytes, -v) unlimited
16
file locks                      (-x) unlimited
Copied!
How do I request network access for my app?
You will need to modify your dxapp.json file so that the key access.network is a list of allowed domain names. You can use "*" to indicate that you want access to everything. The following excerpt gives access (redundantly) to everything, github, and Google.
1
{ "access": {
2
    "network": ["*", "github.com", "google.com"]
3
  },
Copied!
How do I parallelize my app?
You can have your app launch a child job on another machine in the cloud by calling an entry point that you have specified in the bash or Python script that runs your app. To add entry points to your code, you can either generate the parallelized code template using dx-app-wizard (provided in the SDK) to get you started, or add them manually (see the code examples below). A more in-depth tutorial can be found here.
Through a Bash App​
1
# Anything outside the function declarations is always run
2
​
3
myfunc() {
4
        echo $myinput
5
}
6
​
7
main() {
8
        # main gets run when you run the app/applet
9
​
10
        # The following line creates a new job running "myfunc" which
11
        # will receive an input variable $myinput set to "hello world"
12
​
13
        dx-jobutil-new-job myfunc -imyinput='hello world'
14
}
Copied!
Through a Python App****
1
import dxpy
2
​
3
@dxpy.entry_point("myfunc")
4
def myfunc(myinput):
5
    print(myinput)
6
​
7
@dxpy.entry_point("main")
8
def main():
9
    # main gets run when you run the app/applet
10
​
11
    # The following line creates a new job running "myfunc" which
12
    # will receive an input variable myinput set to "hello world"
13
​
14
    dxpy.new_dxjob(fn_input={ "myinput": "hello world" }, fn_name="myfunc")
15
​
16
# The following line will call the appropriate entry point.
17
dxpy.run()
Copied!
What are DNAnexus links, and how are they different from using the data object IDs?
DNAnexus links are JSON hashes containing a data object ID (or, optionally, a project ID as well). The following two hashes are both valid DNAnexus links (the second is called an extended DNAnexus link).
1
{
2
    "$dnanexus_link": "file-B37v04Q7z11654j7gjj000F8"
3
}
4
{
5
    "$dnanexus_link": {
6
        "project": "project-B2f3Pz87z115j6K7yb40001V",
7
        "id": "file-B37v04Q7z11654j7gjj000F8"
8
    }
9
}
Copied!
There are two contexts in which you will want to use DNAnexus links instead of string IDs: 1. JSON details of a data object 2. Job input/output
JSON details
Before closing a data object, you have the option to include non-extended DNAnexus links (i.e. ID-only links) in the details of the object. This allows you to link to an existing data object (e.g. a reference genome) that was used to compute the new data object (e.g. a reference genome indexed for a particular mapper like BWA). Doing so allows you to search your data objects by what objects they link to.
How do they affect cloning (copying between projects)?
Any linked data objects that are also hidden will be copied and moved along with their parent data object.
Job input/output
If an app(let) requires an input or output to be of a data object class (e.g. file), then the value in the input/output JSON hash must be given as a DNAnexus link.
There are convenient utility functions in some of the client libraries and command-line tools so that you don't usually have to formulate the hash yourself.
How do they affect job execution?
Before a job created from running an app or applet (origin or master job) will be started, data objects found as DNAnexus links in its input hash must be closed. Thus, if you run an app on an open file, it will not run until the file has been closed. The input data objects will then be cloned into the new job's temporary workspace right before the job starts running.
Similarly, if an origin or master job has DNAnexus links in its output hash, then those referenced objects must be closed to be cloned as output into the parent container. If they are open when all jobs in the tree have finished running, then the platform will fail the job. If there are any closing objects, the platform will wait until they finish closing before cloning them as output and marking the job as done.
What are job-based object references (JBORs), and how can I use them when running apps?
Job-based object references (JBOR) are JSON hashes with two pieces of information:
a job ID (under the key "job")
the name of an output the job is expected to provide (under the key "field")
They can be used in place of an input value to a new job or an output value of an existing job. Once the referenced job finishes successfully, the JBOR will be replaced with the value found in the referenced job's output. If the referenced job fails or does not provide the requested output, then the job waiting for the value will also fail.
JSON Syntax
1
{
2
    "job": "job-xxxx",
3
    "field": "ref_output_field"
4
}
Copied!
UI
Inside a single workflow, you can use the output of a job as the input of another job by dragging the output of a job to the input of another.
Command line
1
$ dx run someapp -iinput=job-xxxx:ref_output_field
Copied!
For more information on how to use JBORs when writing apps, see the Sample Code page.
How do I view the stdout/stderr and any other logs for a job?
Through Web UI
1.Navigate to the project in which you ran the job
2.Click on the Monitor tab
3.Click on the name of the top-level job
4.Click on the job for which you want the logs
Through Command-Line
1
$ dx watch job-xxxx
Copied!
Where can I find example code for writing applications for the platform?
There are many helpful code snippets to be found on the Sample Code page, listed by programming language. Also, check the Developer Applets public project (platform login required) for working applets (with source) for many common bioinformatics tools. You can use dx get to download and explore the applet source code.
A GitHub repo containing these applets can also be found here.