Developing Apps and Applets
Applets and apps are both executables in the platform that you can run in the cloud. You can mix and match them in the same workflow. They can also both be specified to require special permissions to the project context or to the user's other projects. Here's how they're different:
| Applets | Apps |
Model | can be used as scripts for manipulating data, creating proprietary analysis pipelines, or testing versions before publishing an app; they are easy to create and revise | are general-purpose tools of interest to the community at large that usually strive for compatibility, reproducibility, and robustness |
Platform Representation | are data objects residing in projects | are created from applets and reside in separate data containers outside of users' projects; there is one container per version of an app |
Input/Output Specifications | can be created with no input or output specifications; this means that it can be run with any input, and it makes no guarantees about what it returns | must have input and output specifications so that they behave predictably and can be used compatibly with other apps |
Sharing | are shared by sharing the project in which they live, and are otherwise completely private | are shared by publishing the app to a customizable list of authorized users that can easily discover (via the website or command-line), install, and run it |
Open/Closed Source | expose their source code and attached resources to anyone who has VIEW permissions to their project | can hide their source code and attached resources so that only the app's developers can access them |
Naming | can be given any name | belong to a global namespace, and the first time someone creates an app with a particular name, it is reserved for that user and any others that they designate as developers |
Versioning | each revision has a permanent unique ID, recorded in any job started and any data produced | are published with semantic version numbers (xx.yy.zz), with different versions automatically archived and accessible to platform users |
If you are new to writing applications for the DNAnexus platform, start by building applets. They are easier to iterate quickly and share directly with collaborators in your project. Keep in mind that as long as you write your applet to have both an input and output specification (which we recommend as a good coding practice in general), it can always be built to become an app later.
If you haven't already, you may want to take a look at the Intro to Building Apps tutorial, which walks you through creating an applet that takes in a file and outputs another file.
A brief overview of the steps involved:
- 1.Run dx-app-wizard:
- Pick some app name (e.g.
appname) - Specify any input files or other parameters needed for the executable
- Specify any output files or values that the executable generates
- Pick bash as the language
- Pick the basic template
- 2.Place the executable you want to run into the
appname/resourcesdirectory. - 3.Add a line in the
.shfile in theappname/srcdirectory to run the executable on the file with any parameters received from the input. (Use the lines generated by the app wizard for automatically downloading any file input and uploading file output.) - 4.Run either
dx buildordx build --create-app(depending on whether you want to build an applet or an app).
There are a few different options, depending on where the software can be found and if you would like it to be reproducible.
- 1.If the software you need is available as an APT package in Ubuntu , you can edit the
dxapp.jsonfile to specify it (and, optionally, the version you want). The following JSON excerpt shows you how to request APT software packages; in this case, Java, R, and the samtools packages have been requested and will be available when the app is run.{ "runSpec": {"execDepends": [{"name": "openjdk-6-jre", "version": "6b24-1.11.1-4ubuntu2"},{"name": "r-base-core"},{"name": "samtools"},Note that the APT "Recommends" of the packages are not installed by default. You can simulate this behavior on your local Ubuntu machine with--no-install-recommendsoption toapt-get install. - 2.If the software resides in a globally accessible location (e.g. a git repository hosted on GitHub), you must also request network access to the server hosting it. The following JSON excerpt shows how to request and automatically build a repository hosted on GitHub. (Alternatively, you can also just request access to a particular host and perform the download and build steps manually as part of your app.){ "access": {"network": ["github.com"]},"runSpec": {"execDepends": [{"name": "dx-toolkit","package_manager": "git","url": "[email protected]:dnanexus/dx-toolkit.git","tag": "master","build_commands": "make install DESTDIR=/ PREFIX=/opt/dnanexus"},
- 3.If it is software that you already have and would like to upload and install as part of the app, then you should place any necessary files in the
resourcesdirectory of your app before runningdx build. The build tool will compress and package up the contents of that directory as part of your app, and, when it is run, it will be automatically downloaded and extracted into the root directory/. Your first steps in the code of your app should then be to perform any build or installation commands necessary. - 4.You can build an Asset Bundle, which is compatible with software from both APT and GitHub. It can also be compatible with other software. Please see the Asset Build Process page for more details.
NOTE: The following Java 8 packages are not available via traditional
apt-get; however, we have manually injected them into our APT repo. As a result, you can install them using method #1: the runSpec field of the app's dxapp.json file.openjdk-8-dbgopenjdk-8-demoopenjdk-8-docopenjdk-8-jdkopenjdk-8-jreopenjdk-8-jre-headlessopenjdk-8-jre-jamvmopenjdk-8-jre-zeroopenjdk-8-source
The following languages are fully supported via dx-app-wizard templates, client libraries and tools, and sample code:
If you would like to use a language that is not yet supported, you can still do so by packaging any scripts or files with a Python or bash script that will be responsible for running your code. See the Intro to Building Apps tutorial for an example of how to package up an arbitrary Linux executable as an applet. If you need to install any extra software dependencies such as Java or R, see [the above answer](/FAQ#How-do-I-install-software-requirements-for-my-app-(e.g.-Java,-R,-samtools)%3F) for more information.
By default, a job will run on a virtual machine with these resources:
- Dual-core x86-64 CPU
- 7.5 GB of RAM
- 400 GB scratch file system
To request a different machine, you will need to edit the
dxapp.json file to specify the instance type for each entry point of your app. Please see the documentation on the Run Specification for the list of available instance types and other details.The following
dxapp.json excerpt shows how to request larger virtual machines for both the main and myEntryPoint entry points of your app; any other entry points not listed in systemRequirements will use the default virtual machine mem2_hdd2_x2.{ "runSpec": {
"systemRequirements": {
"main": {
"instanceType": "mem2_hdd2_x4"
},
"myEntryPoint": {
"instanceType": "mem3_hdd2_x2"
}
},
The default limits, as presented by the output of
ulimit -a, are the following:core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 59461
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 59461
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
You will need to modify your
dxapp.json file so that the key access.network is a list of allowed domain names. You can use "*" to indicate that you want access to everything. The following excerpt gives access (redundantly) to everything, github, and Google.{ "access": {
"network": ["*", "github.com", "google.com"]
},
You can have your app launch a subjob on another machine in the cloud by calling an entry point that you have specified in the bash or Python script that runs your app. To add entry points to your code, you can either generate the parallelized code template using
dx-app-wizard (provided in the SDK) to get you started, or add them manually (see the code examples below). A more in-depth tutorial can be found here.# Anything outside the function declarations is always run
myfunc() {
echo $myinput
}
main() {
# main gets run when you run the app/applet
# The following line creates a new job running "myfunc" which
# will receive an input variable $myinput set to "hello world"
dx-jobutil-new-job myfunc -imyinput='hello world'
}
import dxpy
@dxpy.entry_point("myfunc")
def myfunc(myinput):
print(myinput)
@dxpy.entry_point("main")
def main():
# main gets run when you run the app/applet
# The following line creates a new job running "myfunc" which
# will receive an input variable myinput set to "hello world"
dxpy.new_dxjob(fn_input={ "myinput": "hello world" }, fn_name="myfunc")
# The following line will call the appropriate entry point.
dxpy.run()
DNAnexus links are JSON hashes containing a data object ID (or, optionally, a project ID as well). The following two hashes are both valid DNAnexus links (the second is called an extended DNAnexus link).
{
"$dnanexus_link": "file-B37v04Q7z11654j7gjj000F8"
}
{
"$dnanexus_link": {
"project": "project-B2f3Pz87z115j6K7yb40001V",
"id": "file-B37v04Q7z11654j7gjj000F8"
}
}
There are two contexts in which you will want to use DNAnexus links instead of string IDs: 1. JSON details of a data object 2. Job input/output
Before closing a data object, you have the option to include non-extended DNAnexus links (i.e. ID-only links) in the details of the object. This allows you to link to an existing data object (e.g. a reference genome) that was used to compute the new data object (e.g. a reference genome indexed for a particular mapper like BWA). Doing so allows you to search your data objects by what objects they link to.
How do they affect cloning (copying between projects)?
Any linked data objects that are also hidden will be copied and moved along with their parent data object.
If an app(let) requires an input or output to be of a data object class (e.g. file), then the value in the input/output JSON hash must be given as a DNAnexus link.
There are convenient utility functions in some of the client libraries and command-line tools so that you don't usually have to formulate the hash yourself.
How do they affect job execution?
Before a job created from running an app or applet (origin or master job) will be started, data objects found as DNAnexus links in its input hash must be closed. Thus, if you run an app on an open file, it will not run until the file has been closed. The input data objects will then be cloned into the new job's temporary workspace right before the job starts running.
Similarly, if an origin or master job has DNAnexus links in its output hash, then those referenced objects must be closed to be cloned as output into the parent container. If they are open when all jobs in the tree have finished running, then the platform will fail the job. If there are any closing objects, the platform will wait until they finish closing before cloning them as output and marking the job as done.
Job-based object references (JBOR) are JSON hashes with two pieces of information:
- a job ID (under the key "job")
- the name of an output the job is expected to provide (under the key "field")
They can be used in place of an input value to a new job or an output value of an existing job. Once the referenced job finishes successfully, the JBOR will be replaced with the value found in the referenced job's output. If the referenced job fails or does not provide the requested output, then the job waiting for the value will also fail.
JSON Syntax
{
"job": "job-xxxx",
"field": "ref_output_field"
}
UI
Inside a single workflow, you can use the output of a job as the input of another job by dragging the output of a job to the input of another.
Command line
$ dx run someapp -iinput=job-xxxx:ref_output_field
- 1.Navigate to the project in which you ran the job
- 2.Click on the Monitor tab
- 3.Click on the name of the top-level job
- 4.Click on the job for which you want the logs
$ dx watch job-xxxx
There are many helpful code snippets to be found on the Developer Tutorials page, listed by programming language.
If you would like to see more example code, you can use the
dx get command to reconstruct and download the source directory of open-source apps (e.g. dx get app-cloud_workstation). You can find open-source apps with the command below$ dx api system findApps '{"describe":{"fields":{"openSource": true, "name": true}}}'| \
jq '.results|.[]|select(.describe.openSource)|.describe.name'
Last modified 2mo ago