Developing Apps and Applets

What's the difference between an app and an applet?

Applets and apps are both executables in the platform that you can run in the cloud. You can mix and match them in the same workflow. They can also both be specified to require special permissions to the project context or to the user's other projects. Here's how they're different:

Applets

Apps

Model

can be used as scripts for manipulating data, creating proprietary analysis pipelines, or testing versions before publishing an app. They are easy to create and revise

are general-purpose tools of interest to the community at large that strive for compatibility, reproducibility, and robustness

Platform Representation

are data objects residing in projects

are created from applets and reside in separate data containers outside of users' projects. Each version of an app has its own container

Input/Output Specifications

can be created with no input or output specifications. This means that it can be run with any input, and it makes no guarantees about what it returns

must have input and output specifications so that they behave predictably and can be used compatibly with other apps

Sharing

are shared by sharing the project in which they live, and are otherwise private

are shared by publishing the app to a customizable list of authorized users that can easily discover (via the website or command-line), install, and run it

Open/Closed Source

expose their source code and attached resources to anyone who has VIEW permissions to their project

can hide their source code and attached resources so that only the app's developers can access them

Naming

can be given any name

belong to a global namespace, and the first time someone creates an app with a particular name, it is reserved for that user and any others that they choose as developers

Versioning

each revision has a permanent unique ID, recorded in any job started and any data produced

are published with semantic version numbers (xx.yy.zz), with different versions automatically archived and accessible to platform users

Want to read more? Check out the API documentation for applets and apps.

Should I build an applet or an app?

If you are new to writing applications for the DNAnexus Platform, start by building applets. They are easier to iterate quickly and share directly with collaborators in your project. Keep in mind that as long as you write your applet to have both an input and output specification (which we recommend as a good coding practice in general), it can always be built to become an app later.

How do I package a Linux executable into an app(let)?

If you haven't already, you may want to take a look at the Intro to Building Apps tutorial, which walks you through creating an applet that takes in a file and outputs another file.

A brief overview of the steps involved:

Run dx-app-wizard:
- Provide an app name
- Specify any input files or other parameters needed for the executable
- Specify any output files or values that the executable generates
- Pick bash as the language
- Pick the basic template
Place the executable you want to run into the appname/resources directory.
Add a line in the .sh file in the appname/src directory to run the executable on the file with any parameters received from the input. (Use the lines generated by the app wizard for automatically downloading any file input and uploading file output.)
Run either dx build or dx build --create-app (depending on whether you want to build an applet or an app).

How do I install software requirements for my app such as Java, R, SAMtools?

The following options are available, depending on where the software can be found and if you would like it to be reproducible.

If the software you need is available as an APT package in Ubuntu, you can edit the dxapp.json file to specify it (and, optionally, the version you want). The following JSON excerpt shows you how to request APT software packages. In this case, Java, R, and the SAMtools packages have been requested and will be available when the app is run.
```
{ "runSpec": {
"execDepends": [
    {"name": "openjdk-6-jre", "version": "6b24-1.11.1-4ubuntu2"},
    {"name": "r-base-core"},
    {"name": "samtools"},
```
The APT "Recommends" for these packages are not installed by default. To simulate this behavior on your local Ubuntu machine, use the --no-install-recommends option with apt-get install.
If the software resides in a globally accessible location, such as a git repository hosted on GitHub, you must also request network access to the server hosting it. The following JSON excerpt shows how to request and automatically build a repository hosted on GitHub. (Alternatively, you can also just request access to a particular host and perform the download and build steps manually as part of your app.)
```
{ "access": {
"network": ["github.com"]
},
"runSpec": {
"execDepends": [
    {"name": "dx-toolkit",
     "package_manager": "git",
     "url": "[email protected]:dnanexus/dx-toolkit.git",
     "tag": "master",
     "build_commands": "make install DESTDIR=/ PREFIX=/opt/dnanexus"
    },
```
If it is software that you already have and would like to upload and install as part of the app, then you should place any necessary files in the resources directory of your app before running dx build. The build tool will compress and package up the contents of that directory as part of your app, and, when it is run, it will be automatically downloaded and extracted into the root directory /. Your first steps in the code of your app should then be to perform any build or installation commands necessary.
You can build an Asset Bundle, which is compatible with software from both APT and GitHub. It can also be compatible with other software. See the Asset Build Process page for more details.

NOTE: The following Java 8 packages are not available via traditional apt-get. However, we have manually injected them into our APT repository. As a result, you can install them using method #1: the runSpec field of the app's dxapp.json file.

openjdk-8-dbg
openjdk-8-demo
openjdk-8-doc
openjdk-8-jdk
openjdk-8-jre
openjdk-8-jre-headless
openjdk-8-jre-jamvm
openjdk-8-jre-zero
openjdk-8-source

How do I run newer NVIDIA GPU-accelerated software in my app?

If you are trying to use newer NVIDIA GPU-accelerated software in your app, you may find that the NVIDIA GPU kernel-mode driver NVIDIA.ko that is installed outside of the application execution environment does not support the newer CUDA version required by your application. You can request a newer NVIDIA driver using the NVIDIADriver field as described in Run Specification. Alternatively, you can install NVIDIA Forward Compatibility packages to use the newer CUDA version required by your application. To do this, create a DNAnexus asset that includes NVIDIA Forward Compatibility packages, then use the asset in your app as follows:

$ cat dxasset.json
{
  "name": "NVIDIA_forward_compatability_asset",
  "title": "NVIDIA_forward_compatability_asset",
  "description": "NVIDIA_forward_compatability_asset",
  "version": "0.0.1",
  "distribution": "Ubuntu",
  "release": "24.04",
  "instanceType": "mem2_ssd1_gpu_x16"
}

# The Makefile lines below the "all:" heading are prefixed with tabs
$ cat Makefile
SHELL=/bin/bash -e -x -o pipefail
all:
 sudo mv /etc/apt/apt.conf.d/99dnanexus /tmp/
 wget https://developer.download.NVIDIA.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb
 sudo dpkg -i cuda-keyring_1.1-1_all.deb
 sudo apt-get update
 # example below specifies CUDA version 12.5.
 # NVIDIA provides forward compatibility packages for other CUDA versions as well
 sudo DEBIAN_FRONTEND=noninteractive apt-get -y install cuda-toolkit-12-5 cuda-compat-12-5
 sudo bash -c "echo /usr/local/cuda/compat > /etc/ld.so.conf.d/NVIDIA-compat.conf"
 sudo ldconfig
 sudo rm cuda-keyring_1.1-1_all.deb
 sudo mv /tmp/99dnanexus /etc/apt/apt.conf.d/99dnanexus
$ dx build_asset
  ...
* NVIDIA_forward_compatability_asset (create_asset_focal:main) (done) job-xxxx
  testuser 2024-07-01 18:00:00 (runtime 0:17:07)
  Output: asset_bundle = record-GpPQVk80XzzYxZP5Z0J74f7k
 
# Use the asset_bundle created above in the assetDepends field in your app's dxapp.json
$ cd myapp
$ cat dxapp.json
...
 "runSpec": {
  "regionalOptions": {
    "aws:us-east-1": {
      "systemRequirements": {"*": {"instanceType": "mem2_ssd1_gpu_x16"}},
      "assetDepends": [{"id": "record-GpPQVk80XzzYxZP5Z0J74f7k"}]
    }
  }
...

How do I write my app in my favorite programming language?

The following languages are fully supported via dx-app-wizard templates, client libraries and tools, and sample code:

If you would like to use a language that is not yet supported, you can still do so by packaging any scripts or files with a Python or bash script that will be responsible for running your code. See the Intro to Building Apps tutorial for an example of how to package up an arbitrary Linux executable as an applet. If you need to install any extra software dependencies such as Java or R, see How do I install software requirements for my app? for more information.

How do I request more memory/CPU for my app? How do I specify the compute instance type?

By default, a job will run on a virtual machine with these resources:

Dual-core x86-64 CPU
7.5 GB of RAM
400 GB scratch file system

To request a different machine, you will need to edit the dxapp.json file to specify the instance type for each entry point of your app. See the documentation on the Run Specification for the list of available instance types and other details.

The following dxapp.json excerpt shows how to request larger virtual machines for both the main and myEntryPoint entry points of your app. Any other entry points not listed in systemRequirements will use the default virtual machine mem2_hdd2_x2.

{ "runSpec": {
    "systemRequirements": {
      "main": {
        "instanceType": "mem2_hdd2_x4"
      },
      "myEntryPoint": {
        "instanceType": "mem3_hdd2_x2"
      }
    },

What are the default user limits for processes running inside the Linux execution environment?

The default limits, as presented by the output of ulimit -a, are the following:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 59461
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 59461
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

How do I request network access for my app?

You will need to modify your dxapp.json file so that the key access.network is a list of allowed domain names. You can use "*" to indicate that you want access to everything. The following excerpt gives access to everything:

{
  "access": {
    "network": ["*"]
  }
}

The following, meanwhile, limits access to GitHub and Google:

{
  "access": {
      network: [ "github.com", "google.com" ]
  }
}

How do I parallelize my app?

You can parallelize your app by having it launch subjobs on other cloud machines. This is done by calling entry points that you define in your bash or Python script.

To implement parallelization:

Use dx-app-wizard (included in the SDK) to generate a parallelized code template.
Add entry points manually to your code (see examples below).

For a comprehensive walkthrough, see Introduction to Building Apps.

Through a Bash App

# Anything outside the function declarations is always run
myfunc() {
  echo $myinput
}

main() {
  # main gets run when you run the app/applet

  # The following line creates a new job running "myfunc" which
  # will receive an input variable $myinput set to "hello world"

  dx-jobutil-new-job myfunc -imyinput='hello world'
}

Through a Python App

import dxpy

@dxpy.entry_point("myfunc")
def myfunc(myinput):
    print(myinput)

@dxpy.entry_point("main")
def main():
    # main gets run when you run the app/applet

    # The following line creates a new job running "myfunc" which
    # will receive an input variable myinput set to "hello world"

    dxpy.new_dxjob(fn_input={ "myinput": "hello world" }, fn_name="myfunc")

# The following line will call the appropriate entry point.
dxpy.run()

What are DNAnexus links, and how are they different from using the data object IDs?

DNAnexus links are JSON hashes containing a data object ID (or, optionally, a project ID as well). The following two hashes are both valid DNAnexus links (the second is called an extended DNAnexus link).

{
    "$dnanexus_link": "file-B37v04Q7z11654j7gjj000F8"
}
{
    "$dnanexus_link": {
        "project": "project-B2f3Pz87z115j6K7yb40001V",
        "id": "file-B37v04Q7z11654j7gjj000F8"
    }
}

DNAnexus links should be used instead of string IDs in two contexts: 1. JSON details of a data object 2. Job input/output

JSON details

Before closing a data object, you have the option to include non-extended DNAnexus links, specifically ID-only links, in the details of the object. This allows you to link to an existing data object, such as a reference genome, that was used to compute the new data object. A common example is a reference genome indexed for a particular mapper like BWA. Doing so allows you to search your data objects by what objects they link to.

How do they affect cloning (copying between projects)?

Any linked data objects that are also hidden will be copied and moved along with their parent data object.

Job input/output

If an app(let) requires an input or output to be of a data object class, such as a file, then the value in the input/output JSON hash must be given as a DNAnexus link.

Many client libraries and command-line tools provide convenient utility functions, so you don't have to formulate the hash yourself.

How do they affect job execution?

Before a job created from running an app or applet (origin or master job) will be started, data objects found as DNAnexus links in its input hash must be closed. If you run an app on an open file, it will not run until the file has been closed. The input data objects will then be cloned into the new job's temporary workspace right before the job starts running.

Similarly, if an origin or master job has DNAnexus links in its output hash, then those referenced objects must be closed to be cloned as output into the parent container. If they are open when all jobs in the tree have finished running, then the platform will fail the job. If there are any closing objects, the platform will wait until they finish closing before cloning them as output and marking the job as done.

What are job-based object references (JBORs), and how can I use them when running apps?

Job-based object references (JBOR) are JSON hashes with two pieces of information:

a job ID (under the key "job")
the name of an output the job is expected to provide (under the key "field")

They can be used in place of an input value to a new job or an output value of an existing job. Once the referenced job finishes successfully, the JBOR will be replaced with the value found in the referenced job's output. If the referenced job fails or does not provide the requested output, then the job waiting for the value will also fail.

JSON Syntax

{
    "job": "job-xxxx",
    "field": "ref_output_field"
}

In the UI inside a single workflow, you can use the output of a job as the input of another job by dragging the output of a job to the input of another.

Command line

dx run someapp -iinput=job-xxxx:ref_output_field

For more information on how to use JBORs when writing apps, see the Sample Code page.

How do I view the stdout/stderr and any other logs for a job?

Through Web UI

Navigate to the project in which you ran the job
Click on the Monitor tab
Click on the name of the top-level job
Click on the job for which you want the logs

Through Command-Line

dx watch job-xxxx

Where can I find example code for writing applications for the platform?

The Developer Tutorials page contains many helpful code snippets, listed by programming language.

If you would like to see more example code, you can use the dx get command to reconstruct and download the source directory of open-source apps. For instance, you can try dx get app-cloud_workstation. You can find open-source apps with the command below

dx api system findApps \
  '{"describe":{"fields":{"openSource": true, "name": true}}}'| \
  jq '.results|.[]|select(.describe.openSource)|.describe.name'

Last updated 3 days ago

Was this helpful?