Developing Apps and Applets
Last updated
Was this helpful?
Last updated
Was this helpful?
Applets and apps are both executables in the platform that you can run in the cloud. You can mix and match them in the same workflow. They can also both be specified to require special permissions to the project context or to the user's other projects. Here's how they're different:
Applets
Apps
Model
can be used as scripts for manipulating data, creating proprietary analysis pipelines, or testing versions before publishing an app; they are easy to create and revise
are general-purpose tools of interest to the community at large that usually strive for compatibility, reproducibility, and robustness
Platform Representation
are data objects residing in projects
are created from applets and reside in separate data containers outside of users' projects; there is one container per version of an app
Input/Output Specifications
can be created with no input or output specifications; this means that it can be run with any input, and it makes no guarantees about what it returns
must have input and output specifications so that they behave predictably and can be used compatibly with other apps
Sharing
are shared by sharing the project in which they live, and are otherwise completely private
are shared by publishing the app to a customizable list of authorized users that can easily discover (via the website or command-line), install, and run it
Open/Closed Source
expose their source code and attached resources to anyone who has VIEW permissions to their project
can hide their source code and attached resources so that only the app's developers can access them
Naming
can be given any name
belong to a global namespace, and the first time someone creates an app with a particular name, it is reserved for that user and any others that they designate as developers
Versioning
each revision has a permanent unique ID, recorded in any job started and any data produced
are published with semantic version numbers (xx.yy.zz), with different versions automatically archived and accessible to platform users
Want to read more? Check out the API documentation for and .
If you are new to writing applications for the DNAnexus platform, start by building applets. They are easier to iterate quickly and share directly with collaborators in your project. Keep in mind that as long as you write your applet to have both an input and output specification (which we recommend as a good coding practice in general), it can always be built to become an app later.
If you haven't already, you may want to take a look at the tutorial, which walks you through creating an applet that takes in a file and outputs another file.
A brief overview of the steps involved:
Run dx-app-wizard:
Pick some app name (e.g. appname
)
Specify any input files or other parameters needed for the executable
Specify any output files or values that the executable generates
Pick bash as the language
Pick the basic template
Place the executable you want to run into the appname/resources
directory.
Add a line in the .sh
file in the appname/src
directory to run the executable on the file with any parameters received from the input. (Use the lines generated by the app wizard for automatically downloading any file input and uploading file output.)
Run either dx build
or dx build --create-app
(depending on whether you want to build an applet or an app).
There are a few different options, depending on where the software can be found and if you would like it to be reproducible.
Note that the APT "Recommends" of the packages are not installed by default. You can simulate this behavior on your local Ubuntu machine with --no-install-recommends
option to apt-get install
.
If the software resides in a globally accessible location (e.g. a git repository hosted on GitHub), you must also request network access to the server hosting it. The following JSON excerpt shows how to request and automatically build a repository hosted on GitHub. (Alternatively, you can also just request access to a particular host and perform the download and build steps manually as part of your app.)
If it is software that you already have and would like to upload and install as part of the app, then you should place any necessary files in the resources
directory of your app before running dx build
. The build tool will compress and package up the contents of that directory as part of your app, and, when it is run, it will be automatically downloaded and extracted into the root directory /
. Your first steps in the code of your app should then be to perform any build or installation commands necessary.
NOTE: The following Java 8 packages are not available via traditional apt-get
; however, we have manually injected them into our APT repo. As a result, you can install them using method #1: the runSpec
field of the app's dxapp.json
file.
openjdk-8-dbg
openjdk-8-demo
openjdk-8-doc
openjdk-8-jdk
openjdk-8-jre
openjdk-8-jre-headless
openjdk-8-jre-jamvm
openjdk-8-jre-zero
openjdk-8-source
The following languages are fully supported via dx-app-wizard templates, client libraries and tools, and sample code:
By default, a job will run on a virtual machine with these resources:
Dual-core x86-64 CPU
7.5 GB of RAM
400 GB scratch file system
The following dxapp.json
excerpt shows how to request larger virtual machines for both the main
and myEntryPoint
entry points of your app; any other entry points not listed in systemRequirements
will use the default virtual machine mem2_hdd2_x2
.
The default limits, as presented by the output of ulimit -a
, are the following:
You will need to modify your dxapp.json
file so that the key access.network
is a list of allowed domain names. You can use "*" to indicate that you want access to everything. The following excerpt gives access to everything:
The following, meanwhile, limits access to Github and Google:
DNAnexus links are JSON hashes containing a data object ID (or, optionally, a project ID as well). The following two hashes are both valid DNAnexus links (the second is called an extended DNAnexus link).
There are two contexts in which you will want to use DNAnexus links instead of string IDs: 1. JSON details of a data object 2. Job input/output
How do they affect cloning (copying between projects)?
Any linked data objects that are also hidden will be copied and moved along with their parent data object.
There are convenient utility functions in some of the client libraries and command-line tools so that you don't usually have to formulate the hash yourself.
How do they affect job execution?
Similarly, if an origin or master job has DNAnexus links in its output hash, then those referenced objects must be closed to be cloned as output into the parent container. If they are open when all jobs in the tree have finished running, then the platform will fail the job. If there are any closing objects, the platform will wait until they finish closing before cloning them as output and marking the job as done.
Job-based object references (JBOR) are JSON hashes with two pieces of information:
a job ID (under the key "job")
the name of an output the job is expected to provide (under the key "field")
They can be used in place of an input value to a new job or an output value of an existing job. Once the referenced job finishes successfully, the JBOR will be replaced with the value found in the referenced job's output. If the referenced job fails or does not provide the requested output, then the job waiting for the value will also fail.
JSON Syntax
UI
Inside a single workflow, you can use the output of a job as the input of another job by dragging the output of a job to the input of another.
Command line
Navigate to the project in which you ran the job
Click on the Monitor tab
Click on the name of the top-level job
Click on the job for which you want the logs
If you would like to see more example code, you can use the dx get
command to reconstruct and download the source directory of open-source apps (e.g. dx get app-cloud_workstation
). You can find open-source apps with the command below
If the software you need is available as an package in , you can edit the dxapp.json
file to specify it (and, optionally, the version you want). The following JSON excerpt shows you how to request APT software packages; in this case, Java, R, and the packages have been requested and will be available when the app is run.
You can build an , which is compatible with software from both APT and GitHub. It can also be compatible with other software. Please see the page for more details.
If you are trying to use newer NVIDIA GPU-accelerated software in your app, you may find that the NVIDIA GPU kernel-mode driver nvidia.ko
that is installed outside of the application execution environment does not support the newer CUDA version required by your application. You can request a newer NVIDIA driver using the nvidiaDriver
field as described in , or you can install packages to use the newer CUDA version required by your application by creating a DNAnexus asset that includes NVIDIA Forward Compatibility packages, then using the asset in your app as follows:
If you would like to use a language that is not yet supported, you can still do so by packaging any scripts or files with a Python or bash script that will be responsible for running your code. See the tutorial for an example of how to package up an arbitrary Linux executable as an applet. If you need to install any extra software dependencies such as Java or R, see [the above answer](/FAQ#How-do-I-install-software-requirements-for-my-app-(e.g.-Java,-R,-samtools)%3F) for more information.
To request a different machine, you will need to edit the dxapp.json
file to specify the instance type for each entry point of your app. Please see the documentation on the for the list of available instance types and other details.
You can have your app launch a subjob on another machine in the cloud by calling an entry point that you have specified in the bash or Python script that runs your app. To add entry points to your code, you can either generate the parallelized code template using dx-app-wizard
(provided in the SDK) to get you started, or add them manually (see the code examples below). A more in-depth tutorial can be found .
Before closing a data object, you have the option to include non-extended DNAnexus links (i.e. ID-only links) in the of the object. This allows you to link to an existing data object (e.g. a reference genome) that was used to compute the new data object (e.g. a reference genome indexed for a particular mapper like BWA). Doing so allows you to search your data objects by what objects they link to.
If an app(let) to be of a data object class (e.g. file), then the value in the input/output JSON hash must be given as a DNAnexus link.
Before a job created from running an app or applet () will be started, data objects found as DNAnexus links in its input hash must be closed. Thus, if you run an app on an open file, it will not run until the file has been closed. The input data objects will then be cloned into the new job's temporary workspace right before the job starts running.
For more information on how to use JBORs when writing apps, see the page.
There are many helpful code snippets to be found on the page, listed by programming language.