About Projects

On the DNAnexus Platform, a project is first and foremost a means of enabling users to collaborate, by providing them with shared access to specific data and tools.

Projects have a series of features designed to facilitate collaboration, help project members coordinate and organize their work, and ensure appropriate control over both data and tools.

Managing Project Content

A key function of each project is to serve as a shared storehouse of data objects used by project members as they collaborate.

Click on a project's Manage tab to see a list of all the data objects stored in the project. Within the Manage screen, you can browse and manage these objects, with the range of available actions for an object dependent on its type.

The following are four common actions you can perform on objects from within the Manage screen.

Downloading Files

File objects can be directly downloaded from the system. To download a file:

1. Select its row, then click the More Actions button - the "..." icon - at the end of the row showing the file's name.

2. Select "Download" from the list of available actions.

3. Follow the instructions in the modal window that opens.

Getting More Information on Objects

To learn more about an object:

1. Select its row, then click the Show Info Panel button - the "i" icon - in the upper corner of the Manage screen.

2. Select the row showing the name of the object about which you want to know more. An info panel will open on the right, displaying a range of information about the object. This will include its unique ID, as well as metadata about its owner, time of creation, size, tags, properties, and more.

Deleting Objects

To delete an object:

1. Select its row, then click on the More Actions button - the "..." icon - at the end of the row.

2. Select "Delete" from the list of available actions.

3. Follow the instructions in the modal window that opens.

Copying Data to Another Project

1. Select the object or objects you want to copy to a new project, by clicking the box to the left of the name of each object in the objects list.

2. Click the Copy button in the upper right corner of the Manage screen. A modal window will open.

3. Select the project to which you want to copy the object or objects, then select the location within the project to which the objects should be copied.

4. Click the Copy Selected button.

Access and Sharing

Adding Project Members

You can collaborate on the platform by sharing a project with other DNAnexus users. On sharing a project with a user, or group of users in an organization, they become project members, with access at one of the levels described below. Project access can be revoked at any time by a project administrator.

Removing Project Members

To remove a user or org from a project to which you have ADMINISTER access:

1. On the project's Manage screen, click the Share Project button - the "two people" icon - in the top right corner of the page. A modal window will open, showing a list of project members.

2. Find the row showing the user you want to remove from the project.

3. Move your mouse over that row, then click the Remove from Members button at the right end of the row.

Project Access Levels

Project Access Levels: Two Examples

Suppose you have a set of samples sequenced at your lab, and you have a collaborator who's interested in three of the samples. You can upload the data associated with those samples into a new project, then share that new project with your collaborator, granting him or her VIEW access.

Alternatively, suppose that you and your collaborator are working on the same tissue samples, but each of you wants to try a different sequencing process. You can create a new project, then upload your sequenced data to the project. Then grant your collaborator UPLOAD access to the project, allowing him or her to upload his or her data. You'll then both be able to use one another's data to perform downstream analyses.

Restricting Access to Executables

A project admin can configure a project to allow project members to run only specific executables as root executions. The list of permitted executables is set by entering the following command, via the CLI:

$ dx update project project-xxxx --allowed-executables applet-yyyy --allowed-executables workflow-zzzz [...] 

Note that by entering this command, you will overwrite any existing set of permitted executables.

To unset the list, and thus permit project members to run all available executables as root executions, enter the following command:

$ dx update project project-xxxx --unset-allowed-executables 

Note that executables that are called by a permitted executable are permitted to run, even if they are not included in the list.

Project Data Access Controls

Users with ADMINISTER access to a project can restrict the ability of project members to view, copy, delete, and download project data. The project-level boolean flags below provide fine-grained data access control. All data access control flags default tofalse and can be viewed and modified via CLI and platform API. protected, restricted, downloadRestricted, externalUploadRestricted and containsPHI settings can be viewed and modified in the project's Settings web screen as described below.

• protected: If set to true, only project members with ADMINISTER access to the project can can delete project data. Otherwise, project members with ADMINISTER and CONTRIBUTE access can delete project data. This flag corresponds to the Delete Access policy in the project's Settings web interface screen.

• restricted: If set to true,

  • data in this project cannot be cloned to another project

  • data in this project cannot be used as input to a job or an analysis in another project

  • any running app or applet that reads from this project cannot write results to any other project

  • a job running in the project will have singleContext flag set to true irrespective of the singleContext value supplied to /job/new and /executable-xxxx/run, and will only be allowed to use the job's DNAnexus authentication token when issuing requests to the proxied DNAnexus api endpoint within the job. Use of any other authentication token will result in an error.

  This flag corresponds to the Copy Access policy in the project's Settings web interface screen.

• downloadRestricted: If set to true, data in this project cannot be downloaded outside of the platform. For database objects, users would not be able to access the data in the project from outside DNAnexus. This includes read and write operations. This flag corresponds to the Download Access policy in the project's Settings web interface screen.

• databaseUIViewOnly: If set to true, project members with VIEW access will have their access to project databases restricted to the Cohort Browser only. This feature is only available to customers with an Apollo license. Contact DNAnexus Sales for more information.

• containsPHI: If set to true, data in this project is treated as Protected Health Information (PHI), an identifiable health information that can be linked to a specific person. PHI data protection safeguards the confidentiality and integrity of the project data in compliance with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) by imposing additional restrictions documented in PHI Data Protection section. This flag corresponds to the PHI Data Protection setting in the Administration section of a project's Settings web interface screen.

• displayDataProtectionNotice: If set to true, ADMIN users will be able to turn on/off the ability to show a Data Protection Notice to any users accessing the selected project. If the Data Protection Notice feature is enabled for a project, all users, when first accessing the project, will be required to review and confirm their acceptance of a requirement not to egress data from the project. Note that a license is required to use this feature. Contact DNAnexus Sales for more information.

• externalUploadRestricted: If set to true, external file uploads to this project (from outside the job context) are rejected. The creation of Apollo databases, tables, and inserts of data into tables is disallowed from Thrift with a non-job token. This flag corresponds to the External Upload Access policy in the project's Settings web interface screen. A license is required to use this feature. Contact DNAnexus Sales for more information.

• httpsAppIsolatedBrowsing: If set to true, httpsApp access to jobs launched in this project will be wrapped in Isolated Browsing, which will restrict data transfers through the httpsApp job interface. A license is required to use this limited-access feature. Contact DNAnexus Sales for more information.

PHI Data Protection

Protected Health Information, or PHI, is identifiable health information that can be linked to a specific person. On the DNAnexus Platform, PHI Data Protection safeguards the confidentiality and integrity of data in compliance with the Health Insurance Portability and Accountability Act of 1996 (HIPAA).

When PHI Data Protection is enabled for a project, it is subject to the following protective restrictions:

• Data in this project cannot be cloned to other projects that do not have containsPHI set to true

• Any jobs that run in non-PHI projects will not be able to access any data that can only be found in PHI projects

• Job email notifications sent from the project refer to objects by object ID instead of by name, and other information in the notification may be elided. If you receive such a notification, you can view the elided information by logging onto the Platform and opening the notification and accessing it in the Notifications pane, accessible by clicking the "bell" icon at the far right end of the main menu.

• Apollo database access is subject to additional restrictions

• Once PHI Data Protection is activated for a project, it cannot be disabled.

Billing and Charges

On the DNAnexus Platform, running analyses, storing data, and egressing data are billable activities, and always take place within a specific project. Each project is associated with a billing account, to which invoices are sent, covering all billable activities carried out within the project.

Transferring Project Billing Responsibility

Transferring Billing Responsibility to Another User

If you have ADMINISTER access to a project, you can transfer project billing responsibility to another user, by doing the following:

1. On the project's Settings screen, scroll down to the Administration section.

2. Click the Transfer Billing button. A modal window will open.

3. Enter the email address or username of the user to whom you want to transfer billing responsibility for the project.

4. Click Send Transfer Request.

The user will receive an email notification of your request. To finalize the transfer, he or she must log onto the Platform and formally accept it.

Transferring Billing Responsibility to an Org

If you have billable activities access in the org to which you wish to transfer the project, you can change the billing account of the project to the org. To do this, navigate to the project settings page by clicking on the gear icon in the project header. On the project settings page, you can then select which to which billing account the project should be billed.

If you do not have billable activities access in the org you wish to transfer the project to, you will need to transfer the project to a user who does have this access. The recipient will then be able to follow the instructions below to accept a project transfer on behalf of an org.

Cancelling a Transfer of Billing Responsibility

You can cancel a transfer of project billing responsibility, so long as it hasn't yet been formally accepted by the user in question. To do this:

1. Select All Projects from the Projects link in the main menu. Open the project in question. You'll see a Pending Project Ownership Transfer notification at the top of the screen.

2. Click the Cancel Transfer button to cancel the transfer.

Accepting a Transfer Request

When another user initiates a project transfer to you, you’ll receive a project transfer request, via both an email, and a notification accessible by clicking the Notifications button - the "bell" - at the far right end of the main menu.

Note that if you did not already have access to the project being transferred, you'll get VIEW access, and the project will appear in the list on the Projects screen.

To accept the transfer:

1. Open the project. You'll see a Pending Project Ownership Transfer notification in the project header.

2. Click the Accept Transfer button.

3. Select a new billing account for the project from the dropdown of eligible accounts.

Projects with the Auto-Symlink feature Enabled

If the auto-symlink feature has been enabled for a project, billing responsibility for the project cannot be transferred. See this documentation for more on the auto-symlink feature.

Projects with PHI Data Protection Enabled

If a project has PHI Data Protection enabled, it may only be transferred to an org billing account which also has PHI Data Protection enabled.

Sponsored Projects

Ownership of sponsored projects may not be transferred without the sponsorship first being terminated.

Project Sponsorship

A user or org can sponsor the cost of data storage in a project for a fixed term. During the sponsorship period, project members may copy this data to their own projects and store it there, without incurring storage charges.

On setting up the sponsorship, the sponsor sets it end date. The sponsor can change this end date at any time.

Billing responsibility for sponsored projects may not be transferred.

Sponsored projects may not be deleted, without the project sponsor first ending the sponsorship, by changing its end date to a date in the past.

For more information about sponsorship, contact DNAnexus Support.

Learn More

See the Org Management page for detailed information on projects that are billed to an org.

Learn about accessing and working with projects via the CLI:

• Project Navigation

• Path Resolution

Learn about working with projects as a developer:

• Project API Specifications

• Project Permissions and Sharing

The steps below require the DNAnexus SDK. You must download and install it if you have not done so already.

In addition to this Quickstart, there are Developer Tutorials located in the sidebar that go over helpful tips for new users as well. A few of them include:

• Distributed by Chr (sh)

• Parallel by Chr (py)

• R Shiny Example Web App

Step 1. Build a Simple App

Every DNAnexus app starts with 2 files:

• dxapp.json: a file containing the app's metadata: its inputs and outputs, how the app will be run, etc.

• a script that will be executed in the cloud when the app is run

Let's start by creating a file called dxapp.json with the following text:

{ "name": "coolapp",
  "runSpec": {
    "distribution": "Ubuntu",
    "release": "24.04",
    "version": "0",
    "interpreter": "python3",
    "file": "code.py"
  }
}

Above, we've specified the name for our app (coolapp), the type of interpreter (python3) to run our script with, and a path (code.py) to the script that we will create next. ("version":"0") refers to the version of Ubuntu 24.04 application execution environment that supports (python3) interpreter.

Next, we create our script in a file called code.py with the following text:

import dxpy

@dxpy.entry_point('main')
def main(**kwargs):
    print("Hello, DNAnexus!")
    return {}

That's all we need. To build the app, first log in to DNAnexus and start a project with dx login. In the directory with the two files above, run:

$ dx login
 ...
$ dx build -a

Now, run the app and watch the output:

$ dx run coolapp --watch

That's it! You have just made and run your first DNAnexus applet. Applets are lightweight apps that live in your project, and are not visible in the App Library. When you typed dx run, the app ran on its own Linux instance in the cloud. You have exclusive, secure access to the CPU, storage, and memory on the instance. The DNAnexus API lets your app read and write data on the Platform, as well as launch other apps.

The app is now available in the DNAnexus web interface, as part of the project that you started. It can be configured and run in the Workflow Builder, or shared with other users by sharing the project.

Step 2. Run BLAST

Next, we'll make our app do something a bit more interesting: take in two files with FASTA-formatted DNA, run the BLAST tool to compare them, and output the result.

In the cloud, your app will run on Ubuntu Linux 24.04, where BLAST is available as an APT package, ncbi-blast+. You can request that the DNAnexus execution environment install it before your script is run by listing ncbi-blast+ in the execDepends field of your dxapp.json like this:

{ "name": "coolapp",
  "runSpec": {
    "distribution": "Ubuntu",
    "release": "24.04",
    "version": "0",
    "interpreter": "python3",
    "file": "code.py",
    "execDepends": [ {"name": "ncbi-blast+"} ]
  }
}

Next, let's update code.py to run BLAST:

import dxpy, subprocess

@dxpy.entry_point('main')
def main(seq1, seq2):
    dxpy.download_dxfile(seq1, "seq1.fasta")
    dxpy.download_dxfile(seq2, "seq2.fasta")

    subprocess.call("blastn -query seq1.fasta -subject seq2.fasta > report.txt", shell=True)

    report = dxpy.upload_local_file("report.txt")
    return {"blast_result": report}

We're now ready to rebuild the app and test it on some real data. You can use some demo inputs available in the Demo Data project, or you can upload your own data with dx upload or via the website. If you use the Demo Data inputs, make sure the project you are running your app in is the same region as the Demo Data project.

Rebuild the app with dx build -a, and run it like this:

$ dx run coolapp -i seq1="Demo Data:/Developer Quickstart/NC_000868.fasta" -i seq2="Demo Data:/Developer Quickstart/NC_001422.fasta" --watch

Once the job is done, you can examine the output with dx head report.txt, download it with dx download, or view it on the website.

Step 3. Provide an Input/Output Spec

Workflows are a powerful way to visually connect, configure, and run multiple apps in pipelines. To add our app to a workflow and be able to connect its inputs and/or outputs to other apps, our app will need both input and output specifications. Let's update our dxapp.json as follows:

{
  "name": "coolapp",
  "runSpec": {
    "distribution": "Ubuntu",
    "release": "24.04",
    "version": "0",
    "interpreter": "python3",
    "file": "code.py",
    "execDepends": [ {"name": "ncbi-blast+"} ]
  },
  "inputSpec": [
    {"name": "seq1", "class": "file"},
    {"name": "seq2", "class": "file"}
  ],
  "outputSpec": [
    {"name": "blast_result", "class": "file"}
  ]
}

Rebuild the app with dx build -a. You can run it in the same way as before, but now we can add the applet to a workflow. Click "New Workflow" while looking at your project on the website, and click on coolapp once to add it to the workflow. You'll see inputs and outputs appear on the workflow stage which can be connected to other stages in the workflow.

Also, if you now go back to the command line and run dx run coolapp with no input arguments, it will prompt you for the input values for seq1 and seq2.

Step 4. Configure App Settings

In addition to specifying input files, the I/O specification can also be used to configure settings that we want the app to use. For example, we can configure the E-value setting and other BLAST settings with this code and dxapp.json:

code.py

import dxpy, subprocess

@dxpy.entry_point('main')
def main(seq1, seq2, evalue, blast_args):
    dxpy.download_dxfile(seq1, "seq1.fasta")
    dxpy.download_dxfile(seq2, "seq2.fasta")

    command = "blastn -query seq1.fasta -subject seq2.fasta -evalue {e} {args} > report.txt".format(e=evalue, args=blast_args)
    subprocess.call(command, shell=True)

    report = dxpy.upload_local_file("report.txt")
    return {"blast_result": report}

dxapp.json

{
  "name": "coolapp",
  "runSpec": {
    "distribution": "Ubuntu",
    "release": "24.04",
    "version": "0",
    "interpreter": "python3",
    "file": "code.py",
    "execDepends": [ {"name": "ncbi-blast+"} ]
  },
  "inputSpec": [
    {"name": "seq1", "class": "file"},
    {"name": "seq2", "class": "file"},
    {"name": "evalue", "class": "float", "default": 0.01},
    {"name": "blast_args", "class": "string", "default": ""}
  ],
  "outputSpec": [
    {"name": "blast_result", "class": "file"}
  ]
}

Rebuild the app again and add it in the workflow builder. You should now see the evalue and blast_args settings available when you click the gear button on the stage. After building and configuring a workflow, you can run the workflow itself with dx run workflowname.

Step 5. Use SDK Tools

One of the utilities provided in the SDK is dx-app-wizard. This tool will prompt you with a series of questions with which it will create the basic files needed for a new app. It also gives you the option of writing your app as a bash shell script instead of Python. Just run dx-app-wizard to try it out.

Learn More

For additional information and examples of how to run jobs using the CLI, Chapter 5 of this reference guide may be useful. Note that this material is not a part of the official DNAnexus documentation and is for reference only.

Developers new to the DNAnexus platform may find it easier to learn by doing. This page contains a collection of simple tutorials and examples intended to showcase common tasks and methodologies when creating an app(let) on the DNAnexus platform. After reading through the tutorials and examples you should be able to develop app(let)s that:

• Run efficiently: make use of cloud computing methodologies.

• Are easy to debug: let developers understand and resolve issues.

• Use the scale of the cloud: take advantage of the DNAnexus platform’s flexibility

• Are easy to use: reduce support and enable collaboration.

If it’s your first time developing an app(let) be sure to read through the Getting started series. This series will introduce terms and concepts that tutorials and examples will build upon.

These tutorials are not meant to show realistic everyday examples, but rather provide a strong starting point for app(let) developers. Tutorials will showcase simple and varied implementations of the SAMtools view command on the DNAnexus platform.

Bash App(let) Tutorials

Bash app(let)s use dx-toolkit’s, our platform SDK, command line interface along with common bashisms to create bioinformatic pipelines in the cloud.

Bash

• Bash Helpers

• Distributed by Chr (sh)

• Distributed by Region (sh)

• Git Dependency

• Mkfifo and dx cat

• Parallel by Region (sh)

• Parallel xargs by Chr

• Precompiled Binary

• R Shiny Example Web App

• SAMtools count

• TensorBoard Example Web App

Python App(let) Tutorials

Python app(let)s make of use dx-toolkit’s python implementation along with common python modules such as subprocess to create bioinformatic pipelines in the cloud.

Python

• Dash Example Web App

• Distributed by Region (py)

• Parallel by Chr (py)

• Parallel by Region (py)

• Pysam

Web App(let) Tutorials

To create a web applet, you will need access to Titan or Apollo features Web applets can be made as either python or bash applets, the only difference is that they will launch some kind of web server and expose port 443 (for HTTPS) to allow a user to interact with that web application through a web browser.

Web_app

• Dash Example Web App

• R Shiny Example Web App

• TensorBoard Example Web App

Concurrent Computing Tutorials

A bit of terminology before we start discussing parallel and distributed computing paradigms on the DNAnexus Platform.

There are many definitions and approaches to tackling the concept of parallelization and distributing workloads in the cloud (Here’s a particularly helpful Stack Exchange post on the subject). To help make our documentation easier to understand, when discussing concurrent computing paradigms we’ll refer to:

• Parallel: Using multiple threads or logical cores to concurrently process a workload.

• Distributed: Using multiple machines (in our case instances in the cloud) that communicate to concurrently process a workload.

Keep these formal definitions in mind as you read through the tutorials and learn how to compute concurrently on the DNAnexus platform.

Parallel

• Parallel by Chr (py)

• Parallel by Region (py)

• Parallel by Region (sh)

• Parallel xargs by Chr

Distributed

• Distributed by Chr (sh)

• Distributed by Region (py)

• Distributed by Region (sh)

Step 1. Create Your First Project

On the DNAnexus Platform, all data is stored within projects. So before you upload, browse, or analyze any data, you must create a project to house that data.

To create a project:

• Select All Projects from the Projects link in the main menu. This will take you to the Projects page.

• Click the New Project button in the top right corner of the Projects page. The New Project wizard will open in a modal window.

• In the Project Name field, enter a name for your project.

• In the More Info section, you can enter Tags or custom-defined Properties to make it easier to find this project later, and organize it and other projects. For more information on this topic, see this detailed explanation for more information on tags and properties.

• In the More Info section, you can also enter a Project Summary and/or a Project Description.

• In the Billed To field of the Billing section, choose a billing account to which project charges will be billed. Follow these instructions to set up billing.

• In the Billed To field of the Billing section, choose a cloud region in which project files will be stored and analyses will be run. A default region will be displayed here; it's fine to accept this default. For more on this topic, see this detailed explanation of cloud regions.

• In the Access section, specify which types of users will be able to Copy Data, Delete Data, and Download Data. Default values will be shown here; it's fine to accept the defaults. For more on project access, see this detailed explanation of project access levels. For more on types of users, see this detailed rundown.

• Click Create Project. You'll be taken to the Manage screen for the project. Once you've added data to your project, this is where you'll be able to see and get info on this data, and launch analyses that use it.

Step 2. Add Project Members

Once you've created a project, you can add members by doing the following:

1. From the project's Manage screen, click the Share Project button - the "two people" icon - in the top right corner of the project page.

2. Type the username or the email address of an existing Platform user, or the ID of an org whose members you want to add the project.

3. In the Access pulldown, choose the type of access the user or org will have to the project. For more on this, see this detailed explanation of project access levels.

4. If you don't want the user to receive an email notification on being added to the project, click the Email Notification to "Off."

5. Click the Add User button.

6. Repeat Steps 2-5, for each user you want to add to the project.

7. Click Done when you're finished adding members.

Step 3. Add Data to Your Project

To add data to your project, click the Add button in the top right corner of the project's Manage screen. You'll see three options for adding data:

• Upload Data - Use your web browser to upload data from your computer. Note that if the upload takes a significant amount of time, you'll need to ensure that until it completes, you stay logged into the Platform, and keep your browser window open.

• Add Data from Server - Specify an URL of an accessible server from which the file will be uploaded.

• Copy Data from Project - Copy data from another project on the Platform.

Adding Data to Use in Your First Analysis

To prepare for running your first analysis, as detailed in Steps 4-7, copy in data from the "Demo Data" project:

1. From the project's Manage screen, click the Add button, then select Copy Data from Project.

2. In the Copy Data from Project modal window, open the "Demo Data" project by clicking on its name.

3. Open the "Quickstart" folder. This folder contains two 1000 Genomes project files with the paired-end sequencing reads from chromosome 20 of exome SRR100022: SRR100022_20_1.fq.gz and SRR100022_20_2.fq.gz.

4. Click the box next to the Name header, to select both files.

5. Click Copy to copy the files to your project.

Step 4. Install Apps

Next, install the apps you'll need, to analyze the data you added to the project in Step 3:

1. Select Tools Library from the Tools link in the main menu.

2. A list of available tools will open.

3. Find the BWA-MEM FASTQ Read Mapper in the list and click on its name.

4. A tool detail page will open, a full range of information about the tool, and how to use it.

5. Click the Install button in the upper left part of the screen, under the name of the tool.

6. In the Install App modal, click the Agree and Install button.

7. After the tool has been installed, you'll be returned to the tool detail page.

8. Use your browser's "Back" button to return to the tools list page.

9. Repeat Steps 3-6 to install the FreeBayes Variant Caller.

Step 5. Build a Workflow

Now build workflow using the two apps you've just installed, and configure it to use the data you added to your project in Step 3.

Adding Workflow Steps

A workflow runs tools as part of a preconfigured series of steps. Start building your workflow by adding steps to it:

1. Return to your project's Manage screen. You can do this by using your browser's "Back" button, or by selecting All Projects from the Projects link in the main menu, then clicking on the name of your project in the projects list.

2. Click the Add button in the top right corner of the screen, then select New Workflow from the dropdown. The Workflow Builder will open.

3. In the Workflow Builder, give your new workflow a name. In the upper left corner of the screen, you'll see a field with a placeholder value that begins "Untitled Workflow." Click on the "pencil" icon next to this placeholder name, then enter a name of your choosing.

4. Click the Add a Step button. In the Select a Tool modal window, find the BWA-MEM FASTQ Read Mapper and click the "+" to the left of its name, to add it to your workflow.

5. Repeat Step 4 for the FreeBayes Variant Caller.

6. Close the Select a Tool modal window, by clicking either on the "x" in its upper right corner, or the Close button in its lower right corner. You'll return to the main Workflow Builder screen.

Setting Inputs for Each Step

Set the required inputs for each step by doing the following:

1. To set the required inputs for the first step, start by clicking on the input labeled "Reads [array]" for the BWA-MEM FASTQ Read Mapper. In the Select Data for Reads Input modal window, click the box for the SRR100022_20_1.fq.gz file. Then click the Select button.

2. Since the SRR100022 exome was sequenced using paired-end sequencing, you'll need to provide the right-mates for the first set of reads. Click on the input labeled "Reads (right mates) [array]" for the BWA-MEM FASTQ Read Mapper. Select the SRR100022_20_2.fq.gz file.

3. Click on the input labeled "BWA reference genome index." At the bottom of the modal window that opens, there will be a Suggestions section that includes a link to a folder containing reference genome files. Click on this link, then open the folder named H. Sapiens - GRCh37 - b37 (1000 Genomes Phase I). Select the human_g1k_v37.bwa-index.tar.gz file.

4. Next set the "Sorted mappings [array]" required input for the second step. In the "Output" section for the first step, click on the blue pill labeled "Sorted mappings," then drag it to the second step input labeled "Sorted mappings [array]."

5. Click on the second step input labeled "Genome." In the modal that opens, find the reference genomes folder as in Step 3. Open the folder named H. Sapiens - GRCh37 - b37 (1000 Genomes Phase I). Select the human_g1k_v37.fa.gz file.

Step 6. Launch the Workflow

You're ready to launch your workflow, by doing the following:

1. Click the Start Analysis button at the upper right corner of the Workflow Builder.

2. In the modal window that opens, click the Run as Analysis button.

The BWA-MEM FASTQ Read Mapper will start executing immediately. Once it finishes, the FreeBayes Variant Caller will start, using the Read Mapper's output as an input.

Step 7. Monitor Your Job

Once you've launched your workflow, you'll be taken to your project's Monitor screen. Here, you'll see a list of both current and past analyses run within the project, along with key information about each run.

As your workflow runs, its status will be shown as "In Progress."

Terminating Your Job

If for some reason you need to terminate the run before it completes, find its row in the list on the Monitor screen. In the last column on the right, you'll see a red button labeled Terminate. Click the button to terminate the job. Note that this can take some time. While the job is being terminated, the job's status will show as "Terminating."

Step 8. Access the Results

When your workflow completes, output files will be placed into a new folder in your project, with the same name as the workflow. The folder is accessible by navigating to your project's Manage screen.

Running the Workflow Using the Full SRR100022 Exome

You can run this workflow using the full SRR100022 exome, which is available in the SRR100022 folder, in the "Demo Data" project. Note that because this entails working with a much larger file, running the workflow using the exome data will take longer.

Learn More

See these Key Concepts pages for more in-depth treatments of topics that are covered briefly here:

• Projects

• Apps and Workflows

For a video intro to the Platform, watch this series of short, task-oriented tutorials.

For a more in-depth video intro to the Platform, watch this DNAnexus Platform Essentials video.

Source Code

View full source code on GitHub

Step 1. Download BAM Files

Download input files using the dx-download-all-inputs command. The dx-download-all-inputs command will go through all inputs and download into folders with the pattern /home/dnanexus/in/[VARIABLE]/[file or subfolder with files].

dx-download-all-inputs

Step 2. Create an Output Directory

We create an output directory in preparation for dx-upload-all-outputs DNAnexus command in the (Upload Results)[#Upload Result] section.

mkdir -p out/counts_txt

Step 3. Run SAMtools View

After executing the dx-download-all-inputs command, there are three helper variables created to aid in scripting. For this applet, the input variable name mappings_bam with platform filename my_mappings.bam will have a helper variables:

# [VARIABLE]_path the absolute string path to the file.
$ echo $mappings_bam_path
/home/dnanexus/in/mappings_bam/my_mappings.bam
# [VARIABLE]_prefix the file name minus the longest matching pattern in the dxapp.json file
$ echo $mappings_bam_prefix
my_mappings
# [VARIABLE]_name the file name from the platform
$ echo $mappings_bam_name
my_mappings.bam

We use the bash helper variable mappings_bam_path to reference the location of a file after it has been downloaded using dx-download-all-inputs.

samtools view -c "${mappings_bam_path}" > out/counts_txt/"${mappings_bam_prefix}.txt"

Step 4. Upload Result

We use the dx-upload-all-outputs command to upload data to the platform and specify it as the job’s output. The dx-upload-all-outputs command expects to find file paths matching the pattern /home/dnanexus/out/[VARIABLE]/*. It will upload matching files and then associate them as the output corresponding to [VARIABLE]. In this case, the output is called counts_txt. Earlier we created the folders, and we can now place the outputs there.

dx-upload-all-outputs

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json file’s runSpec.execDepends.

{
...
    "runSpec": {
  	...
      "execDepends": [
        {"name": "samtools"}
      ]
    }
...
}

For additional information, see execDepends

Entry Points

Distributed bash-interpreter apps use bash functions to declare entry points. This app has the following entry points specified as bash functions:

• main

• count_func

• sum_reads

Entry points are executed on a new worker with its own system requirements. The instance type can be set in the dxapp.json file’s runSpec.systemRequirements:

{
  "runSpec": {
    ...
    "systemRequirements": {
      "main": {
        "instanceType": "mem1_ssd1_x4"
      },
      "count_func": {
        "instanceType": "mem1_ssd1_x2"
      },
      "sum_reads": {
        "instanceType": "mem1_ssd1_x4"
      }
    },
    ...
  }
}

main

The main function slices the initial *.bam file and generates an index *.bai if needed. The input *.bam is the sliced into smaller *.bam files containing only reads from canonical chromosomes. First, the main function downloads the BAM file and gets the headers.

  dx download "${mappings_sorted_bam}"
  chromosomes=$(samtools view -H "${mappings_sorted_bam_name}" | grep "\@SQ" | awk -F '\t' '{print $2}' | awk -F ':' '{if ($2 ~ /^chr[0-9XYM]+$|^[0-9XYM]/) {print $2}}')

Sliced *.bam files are uploaded and their file IDs are passed to the count_func entry point using the dx-jobutil-new-job command.

  if [ -z "${mappings_sorted_bai}" ]; then
    samtools index "${mappings_sorted_bam_name}"
  else
    dx download "${mappings_sorted_bai}" -o "${mappings_sorted_bam_name}".bai
  fi

  count_jobs=()
  for chr in $chromosomes; do
    seg_name="${mappings_sorted_bam_prefix}_${chr}".bam
    samtools view -b "${mappings_sorted_bam_name}" "${chr}" > "${seg_name}"
    bam_seg_file=$(dx upload "${seg_name}" --brief)
    count_jobs+=($(dx-jobutil-new-job -isegmentedbam_file="${bam_seg_file}" -ichr="${chr}" count_func))
  done

Outputs from the count_func entry points are referenced as Job Based Object References (JBOR) and used as inputs for the sum_reads entry point.

  for job in "${count_jobs[@]}"; do
    readfiles+=("-ireadfiles=${job}:counts_txt")
  done

  sum_reads_job=$(dx-jobutil-new-job "${readfiles[@]}" -ifilename="${mappings_sorted_bam_prefix}" sum_reads)

The output of the sum_reads entry point is used as the output of the main entry point via JBOR reference using the command dx-jobutil-add-output.

count_func

This entry point downloads and runs the command samtools view -c on the sliced *.bam. The generated counts_txt output file is uploaded as the entry point’s job output via the command dx-jobutil-add-output.

count_func () 
{ 
    echo "Value of segmentedbam_file: '${segmentedbam_file}'";
    echo "Chromosome being counted '${chr}'";
    dx download "${segmentedbam_file}";
    readcount=$(samtools view -c "${segmentedbam_file_name}");
    printf "${chr}:\t%s\n" "${readcount}" > "${segmentedbam_file_prefix}.txt";
    readcount_file=$(dx upload "${segmentedbam_file_prefix}".txt --brief);
    dx-jobutil-add-output counts_txt "${readcount_file}" --class=file
}

sum_reads

The main entry point triggers this sub job, providing the output of count_func as an input. This entry point gathers all the files generated by the count_func jobs and sums them.

This function returns read_sum_file as the entry point output.

sum_reads () 
{ 
    set -e -x -o pipefail;
    printf "Value of read file array %s" "${readfiles[@]}";
    echo "Filename: ${filename}";
    echo "Summing values in files and creating output read file";
    for read_f in "${readfiles[@]}";
    do
        echo "${read_f}";
        dx download "${read_f}" -o - >> chromosome_result.txt;
    done;
    count_file="${filename}_chromosome_count.txt";
    total=$(awk '{s+=$2} END {print s}' chromosome_result.txt);
    echo "Total reads: ${total}" >> "${count_file}";
    readfile_name=$(dx upload "${count_file}" --brief);
    dx-jobutil-add-output read_sum_file "${readfile_name}" --class=file
}

View full source code on GitHub

Entry Points

Distributed bash-interpreter apps use bash functions to declare entry points. Entry points are executed as subjobs on new workers with their own respective system requirements. This app has the following entry points specified as bash functions:

• main

• count_func

• sum_reads

main

The main function takes the initial *.bam, generates an index *.bai if needed, and obtains the list of regions from the *.bam file. Every 10 regions will be sent, as input, to the count_func entry point using dx-jobutil-new-job command.

  regions=$(samtools view -H "${mappings_sorted_bam_name}" | grep "\@SQ" | sed 's/.*SN:\(\S*\)\s.*/\1/')

  echo "Segmenting into regions"
  count_jobs=()
  counter=0
  temparray=()
  for r in $(echo $regions); do
    if [[ "${counter}" -ge 10 ]]; then
      echo "${temparray[@]}"
      count_jobs+=($(dx-jobutil-new-job -ibam_file="${mappings_sorted_bam}" -ibambai_file="${mappings_sorted_bai}" "${temparray[@]}" count_func))
      temparray=()
      counter=0
    fi
    temparray+=("-iregions=${r}") # Here we add to an array of -i<parameter>'s
    counter=$((counter+1))
  done

  if [[ counter -gt 0 ]]; then # Previous loop will miss last iteration  if its < 10
    echo "${temparray[@]}"
    count_jobs+=($(dx-jobutil-new-job -ibam_file="${mappings_sorted_bam}" -ibambai_file="${mappings_sorted_bai}" "${temparray[@]}" count_func))
  fi

Job outputs from the count_func entry point are referenced as Job Based Object References (JBOR) and used as inputs for the sum_reads entry point.

  echo "Merge count files, jobs:"
  echo "${count_jobs[@]}"
  readfiles=()
  for count_job in "${count_jobs[@]}"; do
    readfiles+=("-ireadfiles=${count_job}:counts_txt")
  done
  echo "file name: ${sorted_bamfile_name}"
  echo "Set file, readfile variables:"
  echo "${readfiles[@]}"
  countsfile_job=$(dx-jobutil-new-job -ifilename="${mappings_sorted_bam_prefix}" "${readfiles[@]}" sum_reads)

Job outputs of the sum_reads entry point is used as the output of the main entry point via JBOR reference in the dx-jobutil-add-output command.

  echo "Specifying output file"
  dx-jobutil-add-output counts_txt "${countsfile_job}:read_sum" --class=jobref
}

count_func

This entry point performs a SAMtools count of the 10 regions passed as input. This execution will be run on a new worker. As a result variables from other functions (e.g. main()) will not be accessible here.

Once the output file with counts is created, it is uploaded to the platform and assigned as the entry point’s job output counts_txt via the command dx-jobutil-add-output.

count_func() {

  set -e -x -o pipefail

  echo "Value of bam_file: '${bam_file}'"
  echo "Value of bambai_file: '${bambai_file}'"
  echo "Regions being counted '${regions[@]}'"


  dx-download-all-inputs


  mkdir workspace
  cd workspace || exit
  mv "${bam_file_path}" .
  mv "${bambai_file_path}" .
  outputdir="./out/samtool/count"
  mkdir -p "${outputdir}"
  samtools view -c "${bam_file_name}" "${regions[@]}" >> "${outputdir}/readcounts.txt"


  counts_txt_id=$(dx upload "${outputdir}/readcounts.txt" --brief)
  dx-jobutil-add-output counts_txt "${counts_txt_id}" --class=file
}

sum_reads

The main entry point triggers this subjob, providing the output of count_func as an input JBOR. This entry point gathers all the readcount.txt files generated by the count_func jobs and sums the totals.

This entry point returns read_sum as a JBOR, which is then referenced as job output.

sum_reads() {

  set -e -x -o pipefail
  echo "$filename"

  echo "Value of read file array '${readfiles[@]}'"
  dx-download-all-inputs
  echo "Value of read file path array '${readfiles_path[@]}'"

  echo "Summing values in files"
  readsum=0
  for read_f in "${readfiles_path[@]}"; do
    temp=$(cat "$read_f")
    readsum=$((readsum + temp))
  done

  echo "Total reads: ${readsum}" > "${filename}_counts.txt"

  read_sum_id=$(dx upload "${filename}_counts.txt" --brief)
  dx-jobutil-add-output read_sum "${read_sum_id}" --class=file

In the main function, the output is referenced

  echo "Specifying output file"
  dx-jobutil-add-output counts_txt "${countsfile_job}:read_sum" --class=jobref
}

View full source code on GitHub

This applet performs a basic samtools view -c {bam} command, referred to as “SAMtools count”, on the DNAnexus platform.

Download BAM Files

For bash scripts, inputs to a job execution become environment variables. The inputs from our dxapp.json file are formatted as shown below:

{
  "inputSpec": [
    {
      "name": "mappings_bam",
      "label": "Mapping",
      "class": "file",
      "patterns": ["*.bam"],
      "help": "BAM format file."
    }
  ]
}

The object mappings_bam, a DNAnexus link containing the file ID of that file, will be available as an environmental variable in the applet’s execution. Use the command dx download to download the BAM file. By default, when we download a file, we will keep the filename of the object on the platform.

dx download "${mappings_bam}"

SAMtools Count

Here, we use the bash helper variable mappings_bam_name. For file inputs, the DNAnexus platform creates a bash variable [VARIABLE]_name that holds a string representing the filename of the object on the platform; because we downloaded the file using default parameters, this will be the filename of the object on this worker as well. We use another helper variable, [VARIABLE]_prefix, the filename of the object minus any suffixes specified in the input field patterns. From the input spec above, the only pattern present is '["*.bam"]', so the platform will remove the trailing “.bam” and create the helper variable [VARIABLE]_prefix for our use.

readcount=$(samtools view -c "${mappings_bam_name}")
echo "Total reads: ${readcount}" > "${mappings_bam_prefix}.txt"

Upload Result

Use dx upload command to upload data to the platform. This will upload the file into the job container, a temporary project that holds onto files associated with the job. When running the command dx upload with the flag --brief, the command will return just the file ID.

counts_txt_id=$(dx upload "${mappings_bam_prefix}.txt" --brief)

Note: Job containers are an integral part of the execution process, to learn more see Containers for Execution.

Associate With Output

The output of an applet must be declared before the applet is even built. Looking back to the dxapp.json file, we see the following:

{
  "name": "counts_txt",
  "class": "file",
  "label": "Read count file",
  "patterns": [
    "*.txt"
  ],
  "help": "Output file with Total reads as the first line."
}

We declared a file type output named counts_txt. In the applet script, we must tell the system what file should be associated with the output counts_txt. On job completion, usually at the end of the script, this file will be copied from the temporary job container to the project that launched the job.

dx-jobutil-add-output counts_txt "${counts_txt_id}" --class=file

View full source code on GitHub

This example demonstrates how to run TensorBoard inside a DNAnexus applet.

TensorBoard is a web application used to visualize and inspect what is going on inside TensorFlow training. To use TensorBoard, our training script in TensorFlow needs to include code that saves various data to a log directory where TensorBoard can then find the data to display it.

This example uses an example script from the TensorBoard authors. For more guidance on how to use TensorBoard, check out the tensorflow website (external link).

Creating the web application

The applet code runs a training script, which is placed in resources/home/dnanexus/ to make it available in the current working directory of the worker, and then it starts tensorboard on port 443 (HTTPS).

# Start the training script and put it into the background,
# so the next line of code will run immediately
python mnist_tensorboard_example.py --log_dir LOGS_FOR_TENSORBOARD &

# Run TensorBoard
tensorboard  --logdir LOGS_FOR_TENSORBOARD --host 0.0.0.0 --port 443

We run the training script in the background to start TensorBoard immediately, which will let us see the results while training is still running. This is particularly important for long-running training scripts.

Note that for all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server will keep it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts will not be executed.

As with all web apps, the dxapp.json must include "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose port 443.

Creating an applet on DNAnexus

Build the asset with the libraries first:

dx build_asset tensorflow_asset

Take the record ID it outputs and add it to the dxapp.json for the applet.

"runSpec": {
	...
	"assetDepends": [
    {
      "id": "record-xxxx
    }
  ]
	...
}

Then build the applet

dx build -f tensorboard-web-app
dx run tensorboard-web-app

Once it spins up, you can go to that job’s designated URL based on its job ID, https://job-xxxx.dnanexus.cloud/, to see the result.

View full source code on GitHub

What does this applet do?

This applet performs a basic SAMtools count of alignments present in an input BAM.

Prerequisites

The app must have network access to the hostname where the git repository is located. In this example, access.network is set to:

"access": {
  "network": ["github.com"]
}

To learn more about access and network fields see Execution Environment Reference.

How is the SAMtools dependency added?

SAMtools is cloned and built from the SAMtools GitHub repository. Let’s take a closer look at the dxapp.json file’s runSpec.execDepends property:

  "runSpec": {
 ...
    "execDepends": [
        {
        "name": "htslib",
        "package_manager": "git",
        "url": "https://github.com/samtools/htslib.git",
        "tag": "1.3.1",
        "destdir": "/home/dnanexus"
        },
        {"name": "samtools",
        "package_manager": "git",
        "url": "https://github.com/samtools/samtools.git",
        "tag": "1.3.1",
        "destdir": "/home/dnanexus",
        "build_commands": "make samtools"
        }
    ],
...
  }

The execDepends value is a JSON array of dependencies to resolve before the applet source code is run. In this applet, we specify the following git fetch dependency for htslib and SAMtools. Dependencies are resolved in the order they’re specified. Here we must specify htslib first, before samtools build_commands, due to newer versions of SAMtools depending on htslib. An overview of the each property in the git dependency:

• package_manager - Details the type of dependency and how to resolve. supplementary details.

• url - Must point to the server containing the repository. In this case, a github url.

• tag/branch - Git tag/branch to fetch.

• destdir - Directory on worker to which the git repo is cloned.

• build_commands - If needed, build commands to execute. We know our first dependency, htslib, is built when we build SAMtools; as a result, we only specify “build_commands” for the SAMtools dependency.

Note: build_commands are executed from the destdir; use cd when appropriate.

How is SAMtools called in our src script?

Because we set "destdir": "/home/dnanexus" in our dxapp.json, we know the git repo is cloned to the same directory from which our script will execute. Our example directory’s structure:

├── home
│   ├── dnanexus
│       ├── < app script >
│       ├── htslib
│       ├── samtools
│           ├── < samtools binary >

Our samtools command from the app script is samtools/samtools.

Applet Script

main() {
  set -e -x -o pipefail

  dx download "$mappings_bam"

  count_filename="${mappings_bam_prefix}.txt"
  readcount=$(samtools/samtools view -c "${mappings_bam_name}")
  echo "Total reads: ${readcount}" > "${count_filename}"

  counts_txt=$(dx upload "${count_filename}" --brief)
  dx-jobutil-add-output counts_txt "${counts_txt}" --class=file
}

Note: We could’ve built samtools in a destination within our $PATH or added the binary directory to our $PATH. Keep this in mind for your app(let) development

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json runSpec.execDepends.

  "runSpec": {
    ...
    "execDepends": [
      {"name": "samtools"}
    ]
  }

Debugging

The command set -e -x -o pipefail will assist you in debugging this applet:

• -e causes the shell to immediately exit if a command returns a non-zero exit code.

• -x prints commands as they are executed, which is very useful for tracking the job’s status or pinpointing the exact execution failure.

• -o pipefail makes the return code the first non-zero exit code. (Typically, the return code of pipes is the exit code of the last command, which can create difficult to debug problems.)

  Copy

  set -e -x -o pipefail
  echo "Value of mappings_sorted_bam: '${mappings_sorted_bam}'"
  echo "Value of mappings_sorted_bai: '${mappings_sorted_bai}'"
  
  mkdir workspace
  cd workspace
  dx download "${mappings_sorted_bam}"
  
  if [ -z "$mappings_sorted_bai" ]; then
    samtools index "$mappings_sorted_bam_name"
  else
    dx download "${mappings_sorted_bai}"
  fi

  The *.bai file was an optional job input. You can check for a empty or unset var using the bash built-in test [[ - z ${var}} ]]. You can then download or create a *.bai index as needed.

Parallel Run

Bash’s job control system allows for easy management of multiple processes. In this example, bash commands are run in the background as the maximum job executions are controlled in the foreground. You can place processes in the background using the character & after a command.

  chromosomes=$(samtools view -H "${mappings_sorted_bam_name}" | grep "\@SQ" | awk -F '\t' '{print $2}' | awk -F ':' '{if ($2 ~ /^chr[0-9XYM]+$|^[0-9XYM]/) {print $2}}')

  for chr in $chromosomes; do
    samtools view -b "${mappings_sorted_bam_name}" "${chr}" -o "bam_${chr}.bam"
    echo "bam_${chr}.bam"
  done > bamfiles.txt

  busyproc=0
  while read -r b_file; do
    echo "${b_file}"
    if [[ "${busyproc}" -ge "$(nproc)" ]]; then
      echo Processes hit max
      while [[ "${busyproc}" -gt  0 ]]; do
        wait -n # p_id
        busyproc=$((busyproc-1))
      done
    fi
    samtools view -c "${b_file}"> "count_${b_file%.bam}" &
    busyproc=$((busyproc+1))
  done <bamfiles.txt

  while [[ "${busyproc}" -gt  0 ]]; do
    wait -n # p_id
    busyproc=$((busyproc-1))
  done

Job Output

Once the input bam has been sliced, counted, and summed, the output counts_txt is uploaded using the command dx-upload-all-outputs. The following directory structure required for dx-upload-all-outputs is below:

├── $HOME
│   ├── out
│       ├── < output name in dxapp.json >
│           ├── output file

In your applet, upload all outputs by:

  outputdir="${HOME}/out/counts_txt"
  mkdir -p "${outputdir}"
  cat count* | awk '{sum+=$1} END{print "Total reads = ",sum}' > "${outputdir}/${mappings_sorted_bam_prefix}_count.txt"

  dx-upload-all-outputs

View full source code on GitHub

Creating the web application

After configuring an app with Dash, we start the server on port 443.

app.run_server(host='0.0.0.0', port=443)

Inside the dxapp.json, you would add "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose this port.

Note that for all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server will keep it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts will not be executed.

The rest of these instructions apply to building any applet with dependencies stored in an asset.

Creating an applet on DNAnexus

Source dx-toolkit and log in, then run dx-app-wizard with default options.

Creating the asset

dash-asset specifies all the packages and versions we need. We take these from the Dash installation guide (https://dash.plot.ly/installation\)

pip install dash==0.39.0  # The core dash backend
pip install dash-html-components==0.14.0  # HTML components
pip install dash-core-components==0.44.0  # Supercharged components
pip install dash-table==3.6.0  # Interactive DataTable component (new!)
pip install dash-daq==0.1.0  # DAQ components (newly open-sourced!)

We put these into dash-asset/dxasset.json:

{
  ...
  "execDepends": [
    {"name": "dash", "version":"0.39.0", "package_manager": "pip"},
        {"name": "dash-html-components", "version":"0.14.0", "package_manager": "pip"},
        {"name": "dash-core-components", "version":"0.44.0", "package_manager": "pip"},
        {"name": "dash-table", "version":"3.6.0", "package_manager": "pip"},
        {"name": "dash-daq", "version":"0.1.0", "package_manager": "pip"}
  ],
    ...
}

Build the asset:

dx build_asset dash-asset

Use the asset from the applet

Add this asset to the applet’s dxapp.json:

"runSpec": {
    ...
    "assetDepends": [
    {
      "id": "record-xxxx
    }
  ]
    ...
}

Build the applet

Now build and run the applet itself:

dx build -f dash-web-app
dx run dash-web-app

You can always use dx ssh job-xxxx to ssh into the worker and inspect what’s going on or experiment with quick changes Then go to that job’s special URL https://job-xxxx.dnanexus.cloud/ and see the result!

Optional local testing

The main code is in dash-web-app/resources/home/dnanexus/my_app.py with a local launcher script called local_test.py in the same folder. This allows us to launch the same core code in the applet locally to quickly iterate. This is optional because you can also do all testing on the platform itself.

Install locally the same libraries listed above.

To launch the web app locally:

cd dash-web-app/resources/home/dnanexus/
python3 local_test.py

Once it spins up, you can go to that job’s designated URL based on its job ID, https://job-xxxx.dnanexus.cloud/, to see the result.

View full source code on GitHub

Precompiling a Binary

In this applet, the SAMtools binary was precompiled on an Ubuntu machine. A user can do this compilation on an Ubuntu machine of their own, or they can utilize the Cloud Workstation app to build and compile a binary. On the Cloud Workstation, the user can download the SAMtools source code and compile it in the worker environment, ensuring that the binary will run on future workers.

See Cloud Workstation in the App library for more information.

Resources Directory

The SAMtools precompiled binary is placed in the <applet dir>/resources/ directory. Any files found in the resources/ directory will be packaged, uploaded to the platform, and then unpackaged in the root directory \ of the worker. In our case, the resources/ dir is structured as follows:

├── Applet dir
│   ├── src
│   ├── dxapp.json
│   ├── resources
│       ├── usr
│           ├── bin
│               ├── < samtools binary >

When this applet is run on a worker, the resources/ directory will be placed in the worker’s root directory /:

/
├── usr
│   ├── bin
│       ├── < samtools binary >
├── home
│   ├── dnanexus
│   	├── applet script

We are able to access the SAMtools command because the respective binary is visible from the default $PATH variable. The directory /usr/bin/ is part of the $PATH variable, so in our script we can reference the samtools command directly:

samtools view -c "${mappings_bam_name}" > "${mappings_bam_prefix}.txt"\

View full source code on GitHub

This applet performs a SAMtools count on an input file while minimizing disk usage. For additional details on using FIFO (named pipes) special files, run the command man fifo in your shell.Warning: Named pipes require BOTH a stdin and stdout or they will block a process. In these examples, we place incomplete named pipes in background processes so the foreground script process does not block.

To approach this use case, let’s focus on what we want our applet to do:

1. Stream the BAM file from the platform to a worker.

2. As the BAM is streamed, count the number of reads present.

3. Output the result into a file.

4. Stream the result file to the platform.

Stream BAM file from the platform to a worker

First, we establish a named pipe on the worker. Then, we stream to stdin of the named pipe and download the file as a stream from the platform using dx cat.

First, we establish a named pipe on the worker. Then, we stream to stdin of the named pipe and download the file as a stream from the platform using dx cat.

  mkdir workspace
  mappings_fifo_path="workspace/${mappings_bam_name}"
  mkfifo "${mappings_fifo_path}" # FIFO file is created
  dx cat "${mappings_bam}" > "${mappings_fifo_path}" &
  input_pid="$!"

Output BAM file read count

Now that we have created our FIFO special file representing the streamed BAM, we can just call the samtools command as we normally would. The samtools command reading the BAM would provide our BAM FIFO file with a stdout. However, keep in mind that we want to stream the output back to the platform. We must create a named pipe representing our output file too.

  mkdir -p ./out/counts_txt/

  counts_fifo_path="./out/counts_txt/${mappings_bam_prefix}_counts.txt"

  mkfifo "${counts_fifo_path}" # FIFO file is created, readcount.txt
  samtools view -c "${mappings_fifo_path}" > "${counts_fifo_path}" &
  process_pid="$!"

The directory structure created here (~/out/counts_txt) is required to use the dx-upload-all-outputs command in the next step. All files found in the path ~/out/<output name> will be uploaded to the corresponding <output name> specified in the dxapp.json.

The directory structure created here (~/out/counts_txt) is required to use the dx-upload-all-outputs command in the next step. All files found in the path ~/out/<output name> will be uploaded to the corresponding <output name> specified in the dxapp.json.

Stream the result file to the platform

Currently, we’ve established a stream from the platform, piped the stream into a samtools command, and finally outputting the results to another named pipe. However, our background process is still blocked since we lack a stdout for our output file. Luckily, creating an upload stream to the platform will resolve this.

We can upload as a stream to the platform using the commands dx-upload-all-outputs or dx upload -. Make sure to specify --buffer-size if needed.

We can upload as a stream to the platform using the commands dx-upload-all-outputs or dx upload -. Make sure to specify --buffer-size if needed.

  mkdir -p ./out/counts_txt/

  counts_fifo_path="./out/counts_txt/${mappings_bam_prefix}_counts.txt"

  mkfifo "${counts_fifo_path}" # FIFO file is created, readcount.txt
  samtools view -c "${mappings_fifo_path}" > "${counts_fifo_path}" &
  process_pid="$!"

Note: Alternatively, dx upload - can upload directly from stdin. In this example, we would no longer need to have the directory structure required for dx-upload-all-outputs.Warning: When uploading a file that exists on disk, dx upload is aware of the file size and automatically handles any cloud service provider upload chunk requirements. When uploading as a stream, the file size is not automatically known and dx upload uses default parameters. While these parameters are fine for most use cases, you may need to specify upload part size with the --buffer-size option.

Wait for background processes

Now that our background processes are no longer blocking the rest of the applet’s execution, we simply wait in the foreground for those processes to finish.

  wait -n  # "$input_pid"
  wait -n  # "$process_pid"
  wait -n  # "$upload_pid"

Note: If we didn’t wait the app script would running in the foreground would finish and terminate the job! We wouldn’t want that.

How is the SAMtools dependency provided?

The SAMtools compiled binary is placed directly in the <applet dir>/resources directory. Any files found in the resources/ directory will be uploaded so that they will be present in the worker’s root directory. In our case:

├── Applet dir
│   ├── src
│   ├── dxapp.json
│   ├── resources
│       ├── usr
│           ├── bin
│               ├── < samtools binary >

When this applet is run on a worker, the resources/ folder will be placed in the worker’s root directory /:

/
├── usr
│   ├── bin
│       ├── < samtools binary >
├── home
│   ├── dnanexus

/usr/bin is part of the $PATH variable, so we can reference the samtools command directly in our script as samtools view -c ...

View full source code on GitHub

In order to take full advantage of the scalability that cloud computing offers, our scripts have to implement the correct methodologies. This applet tutorial will:

1. Install SAMtools

2. Download BAM file

3. Count regions in parallel

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json runSpec.execDepends.

  "runSpec": {
    ...
    "execDepends": [
      {"name": "samtools"}
    ]
  }

For additional information, please refer to the execDepends documentation.

Download BAM file

The dxpy.download_all_inputs() function downloads all input files into the /home/dnanexus/in directory. A folder will be created for each input and the file(s) will be downloaded to that directory. For convenience, the dxpy.download_all_inputs function returns a dictionary containing the following keys:

• <var>_path (string): full absolute path to where the file was downloaded.

• <var>_name (string): name of the file, including extention.

• <var>_prefix (string): name of the file minus the longest matching pattern found in the dxapp.json I/O pattern field.

The path, name, and prefix key-value pattern is repeated for all applet file class inputs specified in the dxapp.json. In this example, our dictionary has the following key-value pairs:

{
    mappings_bam_path: [u'/home/dnanexus/in/mappings_bam/SRR504516.bam']
    mappings_bam_name: [u'SRR504516.bam']
    mappings_bam_prefix: [u'SRR504516']
    index_file_path: [u'/home/dnanexus/in/index_file/SRR504516.bam.bai']
    index_file_name: [u'SRR504516.bam.bai']
    index_file_prefix: [u'SRR504516']
}

    inputs = dxpy.download_all_inputs()
    shutil.move(inputs['mappings_bam_path'][0], os.getcwd())

Count Regions in Parallel

Before we can perform our parallel SAMtools count, we must determine the workload for each thread. We arbitrarily set our number of workers to 10 and set the workload per thread to 1 chromosome at a time. There are various ways to achieve multithreaded processing in python. For the sake of simplicity, we use multiprocessing.dummy, a wrapper around Python’s threading module.

    input_bam = inputs['mappings_bam_name'][0]

    bam_to_use = create_index_file(input_bam)
    print("Dir info:")
    print(os.listdir(os.getcwd()))

    regions = parseSAM_header_for_region(bam_to_use)

    view_cmds = [
        create_region_view_cmd(bam_to_use, region)
        for region
        in regions]

    print('Parallel counts')
    t_pools = ThreadPool(10)
    results = t_pools.map(run_cmd, view_cmds)
    t_pools.close()
    t_pools.join()

    verify_pool_status(results)

Each worker creates a string to be called in a subprocess.Popen call. We use the multiprocessing.dummy.Pool.map(<func>, <iterable>) function to call the helper function run_cmd for each string in the iterable of view commands. Because we perform our multithreaded processing using subprocess.Popen, we will not be alerted to any failed processes. We verify our closed workers in the verify_pool_status helper function.

def verify_pool_status(proc_tuples):
    err_msgs = []
    for proc in proc_tuples:
        if proc[2] != 0:
            err_msgs.append(proc[1])
    if err_msgs:
        raise dxpy.exceptions.AppInternalError(b"\n".join(err_msgs))

Important: In this example we use subprocess.Popen to process and verify our results in verify_pool_status. In general, it is considered good practice to use python’s built-in subprocess convenience functions. In this case, subprocess.check_call would achieve the same goal.

Gather Results

Each worker returns a read count of just one region in the BAM file. We sum and output the results as the job output. We use the dx-toolkit python SDK’s dxpy.upload_local_file function to upload and generate a DXFile corresponding to our result file. For python, job outputs have to be a dictionary of key-value pairs, with the keys being job output names as defined in the dxapp.json and the values being the output values for corresponding output classes. For files, the output type is a DXLink. We use the dxpy.dxlink function to generate the appropriate DXLink value.

    resultfn = bam_to_use[:-4] + '_count.txt'
    with open(resultfn, 'w') as f:
        sum_reads = 0
        for res, reg in zip(results, regions):
            read_count = int(res[0])
            sum_reads += read_count
            f.write("Region {0}: {1}\n".format(reg, read_count))
        f.write("Total reads: {0}".format(sum_reads))

    count_file = dxpy.upload_local_file(resultfn)
    output = {}
    output["count_file"] = dxpy.dxlink(count_file)

    return output

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools compiled binary is placed directory in the <applet dir>/resources directory. Any files found in the resources/ directory will be uploaded so that they will be present in the root directory of the worker. In our case:

├── Applet dir
│   ├── src
│   ├── dxapp.json
│   ├── resources
│       ├── usr
│           ├── bin
│               ├── < samtools binary >

When this applet is run on a worker, the resources/ folder will be placed in the worker’s root directory /:

/
├── usr
│   ├── bin
│       ├── < samtools binary >
├── home
│   ├── dnanexus

/usr/bin is part of the $PATH variable, so in our script, we can reference the samtools command directly, as in samtools view -c ...

Parallel Run

Splice BAM

First, we download our BAM file and slice it by canonical chromosome, writing the *bam file names to another file.

In order to split a BAM by regions, we need to have a *.bai index. You can either create an app(let) which takes the *.bai as an input or generate a *.bai in the applet. In this tutorial, we generate the *.bai in the applet, sorting the BAM if necessary.

  dx download "${mappings_bam}"

  indexsuccess=true
  bam_filename="${mappings_bam_name}"
  samtools index "${mappings_bam_name}" || indexsuccess=false
  if [[ $indexsuccess == false ]]; then
    samtools sort -o "${mappings_bam_name}" "${mappings_bam_name}"
    samtools index "${mappings_bam_name}"
    bam_filename="${mappings_bam_name}"
  fi


  chromosomes=$(samtools view -H "${bam_filename}" | grep "\@SQ" | awk -F '\t' '{print $2}' | awk -F ':' '{if ($2 ~ /^chr[0-9XYM]+$|^[0-9XYM]/) {print $2}}')

  for chr in $chromosomes; do
    samtools view -b "${bam_filename}" "${chr}" -o "bam_${chr}."bam
    echo "bam_${chr}.bam"
  done > bamfiles.txt

Xargs SAMtools view

In the previous section, we recorded the name of each sliced BAM file into a record file. Now we will perform a samtools view -c on each slice using the record file as input.

  counts_txt_name="${mappings_bam_prefix}_count.txt"

  sum_reads=$(<bamfiles.txt xargs -I {} samtools view -c $view_options '{}' | awk '{s+=$1} END {print s}')
  echo "Total Count: ${sum_reads}" > "${counts_txt_name}"

Upload results

The results file is uploaded using the standard bash process:

1. Upload a file to the job execution’s container.

2. Provide the DNAnexus link as a job’s output using the script dx-jobutil-add-output <output name>

   Copy

     counts_txt_id=$(dx upload "${counts_txt_name}" --brief)
     dx-jobutil-add-output counts_txt "${counts_txt_id}" --class=file

View full source code on GitHub

Creating the web application

Inside the dxapp.json, you would add "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose this port.

R Shiny needs two scripts, server.R and ui.R, which should be under resources/home/dnanexus/my_app/. When a job starts based on this applet, the resources directory is copied onto the worker, and since the ~/ path on the worker is /home/dnanexus, that means you now have ~/my_app with those two scripts inside.

From the main applet script code.sh, we simply start shiny pointing to ~/my_app repo, serving its mini-application on port 443.

main() {
  R -e "shiny::runApp('~/my_app', host='0.0.0.0', port=443)"
}

Note that for all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server will keep it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts will not be executed.

Modifying this example for your own applet

To make your own applet with R Shiny, simply copy the source code from this example and modify server.R and ui.R inside resources/home/dnanexus/my_app.

How to rebuild the shiny asset

View dxasset.json file

To build the asset, run the dx build_asset command and pass shiny-asset, i.e. the name of the directory holding dxasset.json:

dx build_asset shiny-asset

This will output a record ID record-xxxx that you can then put into the applet’s dxapp.json in place of the existing one:

"runSpec": {
    ...
    "assetDepends": [
    {
      "id": "record-xxxx
    }
  ]
    ...
}

Build the applet

Now build and run the applet itself:

dx build -f dash-web-app
dx run dash-web-app

Once it spins up, you can go to that job’s designated URL based on its job ID, https://job-xxxx.dnanexus.cloud/, to see the result.

View full source code on GitHub

In order to take full advantage of the scalability that cloud computing offers, our scripts have to implement the correct methodologies. This applet tutorial will:

1. Install SAMtools

2. Download BAM file

3. Split workload

4. Count regions in parallel

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json runSpec.execDepends field.

{
  "runSpec": {
    ...
    "execDepends": [
      {"name": "samtools"}
    ]
  }

Download Inputs

This applet downloads all inputs at once using dxpy.download_all_inputs:

inputs = dxpy.download_all_inputs()
# download_all_inputs returns a dictionary that contains mapping from inputs to file locations.
# Additionaly, helper keys, value pairs are added to the dicitonary, similar to bash helper functions
inputs
#     mappings_sorted_bam_path: [u'/home/dnanexus/in/mappings_sorted_bam/SRR504516.bam']
#     mappings_sorted_bam_name: u'SRR504516.bam'
#     mappings_sorted_bam_prefix: u'SRR504516'
#     mappings_sorted_bai_path: u'/home/dnanexus/in/mappings_sorted_bai/SRR504516.bam.bai'
#     mappings_sorted_bai_name: u'SRR504516.bam.bai'
#     mappings_sorted_bai_prefix: u'SRR504516'

Split workload

We process in parallel using the python multiprocessing module using a rather simple pattern shown below:

print("Number of cpus: {0}".format(cpu_count()))  # Get cpu count from multiprocessing
worker_pool = Pool(processes=cpu_count())         # Create a pool of workers, 1 for each core
results = worker_pool.map(run_cmd, collection)    # map run_cmds to a collection
                                                  # Pool.map will handle orchestrating the job
worker_pool.close()
worker_pool.join()  # Make sure to close and join workers when done

This convenient pattern allows you to quickly orchestrate jobs on a worker. For more detailed overview of the multiprocessing module, visit the python docs.

We create several helpers in our applet script to manage our workload. One helper you may have seen before is run_cmd; we use this function to manage or subprocess calls:

def run_cmd(cmd_arr):
    """Run shell command.
    Helper function to simplify the pool.map() call in our parallelization.
    Raises OSError if command specified (index 0 in cmd_arr) isn't valid
    """
    proc = subprocess.Popen(
        cmd_arr,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)
    stdout, stderr = proc.communicate()
    exit_code = proc.returncode
    proc_tuple = (stdout, stderr, exit_code)
    return proc_tuple

Before we can split our workload, we need to know what regions are present in our BAM input file. We handle this initial parsing in the parse_sam_header_for_region function:

def parse_sam_header_for_region(bamfile_path):
    """Helper function to match SN regions contained in SAM header

    Returns:
        regions (list[string]): list of regions in bam header
    """
    header_cmd = ['samtools', 'view', '-H', bamfile_path]
    print('parsing SAM headers:', " ".join(header_cmd))
    headers_str = subprocess.check_output(header_cmd).decode("utf-8")
    rgx = re.compile(r'SN:(\S+)\s')
    regions = rgx.findall(headers_str)
    return regions

Once our workload is split and we’ve started processing, we wait and review the status of each Pool worker. Then, we merge and output our results.

# Write results to file
resultfn = inputs['mappings_sorted_bam_name'][0]
resultfn = (
    resultfn[:-4] + '_count.txt'
    if resultfn.endswith(".bam")
    else resultfn + '_count.txt')
with open(resultfn, 'w') as f:
    sum_reads = 0
    for res, reg in zip(results, regions):
        read_count = int(res[0])
        sum_reads += read_count
        f.write("Region {0}: {1}\n".format(reg, read_count))
    f.write("Total reads: {0}".format(sum_reads))

count_file = dxpy.upload_local_file(resultfn)
output = {}
output["count_file"] = dxpy.dxlink(count_file)
return output

Note: The run_cmd function returns a tuple containing the stdout, stderr, and exit code of the subprocess call. We parse these outputs from our workers to determine whether the run failed or passed.

def verify_pool_status(proc_tuples):
    """
    Helper to verify worker succeeded.

    As failed commands are detected we write the stderr from that command
    to the job_error.json file. This file will be printed to the platform
    job log on App failure.
    """
    all_succeed = True
    err_msgs = []
    for proc in proc_tuples:
        if proc[2] != 0:
            all_succeed = False
            err_msgs.append(proc[1])
    if err_msgs:
        raise dxpy.exceptions.AppInternalError(b"\n".join(err_msgs))

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json runSpec.execDepends.

  "runSpec": {
    ...
    "execDepends": [
      {"name": "samtools"}
    ]
  }

For additional information, please refer to the execDepends documentation .

Entry Points

Distributed python-interpreter apps use python decorators on functions to declare entry points. This app has the following entry points as decorated functions:

• main

• samtoolscount_bam

• combine_files

Entry points are executed on a new worker with their own system requirements. In this example, we split and merge our files on basic mem1_ssd1_x2 instances and perform our own, more intensive, processing step on a mem1_ssd1_x4 instance. Instance type can be set in the dxapp.json runSpec.systemRequirements:

  "runSpec": {
    ...
    "systemRequirements": {
      "main": {
        "instanceType": "mem1_ssd1_x2"
      },
      "samtoolscount_bam": {
        "instanceType": "mem1_ssd1_x4"
      },
      "combine_files": {
        "instanceType": "mem1_ssd1_x2"
      }
    },
    ...
  }

main

The main function scatters by region bins based on user input. If no *.bai file is present, the applet generates an index *.bai.

    regions = parseSAM_header_for_region(filename)
    split_regions = [regions[i:i + region_size]
                     for i in range(0, len(regions), region_size)]

    if not index_file:
        mappings_bam, index_file = create_index_file(filename, mappings_bam)

Regions bins are passed to the samtoolscount_bam entry point using the dxpy.new_dxjob function.

    print('creating subjobs')
    subjobs = [dxpy.new_dxjob(
               fn_input={"region_list": split,
                         "mappings_bam": mappings_bam,
                         "index_file": index_file},
               fn_name="samtoolscount_bam")
               for split in split_regions]

    fileDXLinks = [subjob.get_output_ref("readcount_fileDX")
                   for subjob in subjobs]

Outputs from the samtoolscount_bam entry points are used as inputs for the combine_files entry point. The output of the combine_files entry point is used as the output of the main entry point.

    print('combining outputs')
    postprocess_job = dxpy.new_dxjob(
        fn_input={"countDXlinks": fileDXLinks, "resultfn": filename},
        fn_name="combine_files")

    countDXLink = postprocess_job.get_output_ref("countDXLink")

    output = {}
    output["count_file"] = countDXLink

    return output

samtoolscount_bam

This entry point downloads and creates a samtools view -c command for each region in the input bin. The dictionary returned from dxpy.download_all_inputs() is used to reference input names and paths.

def samtoolscount_bam(region_list, mappings_bam, index_file):
    """Processing function.

    Arguments:
        region_list (list[str]): Regions to count in BAM
        mappings_bam (dict): dxlink to input BAM
        index_file (dict): dxlink to input BAM

    Returns:
        Dictionary containing dxlinks to the uploaded read counts file
    """
    #
    # Download inputs
    # -------------------------------------------------------------------
    # dxpy.download_all_inputs will download all input files into
    # the /home/dnanexus/in directory.  A folder will be created for each
    # input and the file(s) will be download to that directory.
    #
    # In this example our dictionary inputs has the following key, value pairs
    # Note that the values are all list
    #     mappings_bam_path: [u'/home/dnanexus/in/mappings_bam/<bam filename>.bam']
    #     mappings_bam_name: [u'<bam filename>.bam']
    #     mappings_bam_prefix: [u'<bam filename>']
    #     index_file_path: [u'/home/dnanexus/in/index_file/<bam filename>.bam.bai']
    #     index_file_name: [u'<bam filename>.bam.bai']
    #     index_file_prefix: [u'<bam filename>']
    #

    inputs = dxpy.download_all_inputs()

    # SAMtools view command requires the bam and index file to be in the same
    shutil.move(inputs['mappings_bam_path'][0], os.getcwd())
    shutil.move(inputs['index_file_path'][0], os.getcwd())
    input_bam = inputs['mappings_bam_name'][0]

    #
    # Per region perform SAMtools count.
    # --------------------------------------------------------------
    # Output count for regions and return DXLink as job output to
    # allow other entry points to download job output.
    #

    with open('read_count_regions.txt', 'w') as f:
        for region in region_list:
                view_cmd = create_region_view_cmd(input_bam, region)
                region_proc_result = run_cmd(view_cmd)
                region_count = int(region_proc_result[0])
                f.write("Region {0}: {1}\n".format(region, region_count))
    readcountDXFile = dxpy.upload_local_file("read_count_regions.txt")
    readCountDXlink = dxpy.dxlink(readcountDXFile.get_id())

    return {"readcount_fileDX": readCountDXlink}

This entry point returns {"readcount_fileDX": readCountDXlink}, a JBOR referencing an uploaded text file. This approach to scatter-gather stores the results in files and uploads/downloads the information as needed. This approach exaggerates a scatter-gather for tutorial purposes. You’re able to pass types other than file such as int.

combine_files

The main entry point triggers this subjob, providing the output of samtoolscount_bam as an input. This entry point gathers all the files generated by the samtoolscount_bam jobs and sums them.

def combine_files(countDXlinks, resultfn):
    """The 'gather' subjob of the applet.

    Arguments:
        countDXlinks (list[dict]): list of DXlinks to process job output files.
        resultfn (str): Filename to use for job output file.

    Returns:
        DXLink for the main function to return as the job output.

    Note: Only the DXLinks are passed as parameters.
    Subjobs work on a fresh instance so files must be downloaded to the machine
    """
    if resultfn.endswith(".bam"):
        resultfn = resultfn[:-4] + '.txt'

    sum_reads = 0
    with open(resultfn, 'w') as f:
        for i, dxlink in enumerate(countDXlinks):
            dxfile = dxpy.DXFile(dxlink)
            filename = "countfile{0}".format(i)
            dxpy.download_dxfile(dxfile, filename)
            with open(filename, 'r') as fsub:
                for line in fsub:
                    sum_reads += parse_line_for_readcount(line)
                    f.write(line)
        f.write('Total Reads: {0}'.format(sum_reads))

    countDXFile = dxpy.upload_local_file(resultfn)
    countDXlink = dxpy.dxlink(countDXFile.get_id())

    return {"countDXLink": countDXlink}

Important: While the main entry point triggers the processing and gathering entry points, keep in mind the main entry point doesn’t do any heavy lifting or processing. Notice in the .runSpec json above we start with a lightweight instance, scale up for the processing entry point, then finally scale down for the gathering step.

Entry Points

Distributed bash-interpreter apps use bash functions to declare entry points. Entry points are executed as subjobs on new workers with their own respective system requirements. This app has the following entry points specified as bash functions:

• main

• count_func

• sum_reads

main

The main function takes the initial *.bam, generates an index *.bai if needed, and obtains the list of regions from the *.bam file. Every 10 regions will be sent, as input, to the count_func entry point using dx-jobutil-new-job command.

  regions=$(samtools view -H "${mappings_sorted_bam_name}" | grep "\@SQ" | sed 's/.*SN:\(\S*\)\s.*/\1/')

  echo "Segmenting into regions"
  count_jobs=()
  counter=0
  temparray=()
  for r in $(echo $regions); do
    if [[ "${counter}" -ge 10 ]]; then
      echo "${temparray[@]}"
      count_jobs+=($(dx-jobutil-new-job -ibam_file="${mappings_sorted_bam}" -ibambai_file="${mappings_sorted_bai}" "${temparray[@]}" count_func))
      temparray=()
      counter=0
    fi
    temparray+=("-iregions=${r}") # Here we add to an array of -i<parameter>'s
    counter=$((counter+1))
  done

  if [[ counter -gt 0 ]]; then # Previous loop will miss last iteration  if its < 10
    echo "${temparray[@]}"
    count_jobs+=($(dx-jobutil-new-job -ibam_file="${mappings_sorted_bam}" -ibambai_file="${mappings_sorted_bai}" "${temparray[@]}" count_func))
  fi

Job outputs from the count_func entry point are referenced as Job Based Object References (JBOR) and used as inputs for the sum_reads entry point.

  echo "Merge count files, jobs:"
  echo "${count_jobs[@]}"
  readfiles=()
  for count_job in "${count_jobs[@]}"; do
    readfiles+=("-ireadfiles=${count_job}:counts_txt")
  done
  echo "file name: ${sorted_bamfile_name}"
  echo "Set file, readfile variables:"
  echo "${readfiles[@]}"
  countsfile_job=$(dx-jobutil-new-job -ifilename="${mappings_sorted_bam_prefix}" "${readfiles[@]}" sum_reads)

Job outputs of the sum_reads entry point is used as the output of the main entry point via JBOR reference in the dx-jobutil-add-output command.

  echo "Specifying output file"
  dx-jobutil-add-output counts_txt "${countsfile_job}:read_sum" --class=jobref
}

count_func

This entry point performs a SAMtools count of the 10 regions passed as input. This execution will be run on a new worker. As a result variables from other functions (e.g. main()) will not be accessible here.

Once the output file with counts is created, it is uploaded to the platform and assigned as the entry point’s job output counts_txt via the command dx-jobutil-add-output.

count_func() {

  set -e -x -o pipefail

  echo "Value of bam_file: '${bam_file}'"
  echo "Value of bambai_file: '${bambai_file}'"
  echo "Regions being counted '${regions[@]}'"


  dx-download-all-inputs


  mkdir workspace
  cd workspace || exit
  mv "${bam_file_path}" .
  mv "${bambai_file_path}" .
  outputdir="./out/samtool/count"
  mkdir -p "${outputdir}"
  samtools view -c "${bam_file_name}" "${regions[@]}" >> "${outputdir}/readcounts.txt"


  counts_txt_id=$(dx upload "${outputdir}/readcounts.txt" --brief)
  dx-jobutil-add-output counts_txt "${counts_txt_id}" --class=file
}

sum_reads

The main entry point triggers this subjob, providing the output of count_func as an input JBOR. This entry point gathers all the readcount.txt files generated by the count_func jobs and sums the totals.

This entry point returns read_sum as a JBOR, which is then referenced as job output.

sum_reads() {

  set -e -x -o pipefail
  echo "$filename"

  echo "Value of read file array '${readfiles[@]}'"
  dx-download-all-inputs
  echo "Value of read file path array '${readfiles_path[@]}'"

  echo "Summing values in files"
  readsum=0
  for read_f in "${readfiles_path[@]}"; do
    temp=$(cat "$read_f")
    readsum=$((readsum + temp))
  done

  echo "Total reads: ${readsum}" > "${filename}_counts.txt"

  read_sum_id=$(dx upload "${filename}_counts.txt" --brief)
  dx-jobutil-add-output read_sum "${read_sum_id}" --class=file

In the main function, the output is referenced

  echo "Specifying output file"
  dx-jobutil-add-output counts_txt "${countsfile_job}:read_sum" --class=jobref
}

View full source code on GitHub

How is Pysam provided?

Pysam is provided through a pip3 install using the pip3 package manager in the dxapp.json’s runSpec.execDepends property:

{
 "runSpec": {
    ...
    "execDepends": [
      {"name": "pysam",
         "package_manager": "pip3",
         "version": "0.15.4"
      }
    ]
    ...
 }

The execDepends value is a JSON array of dependencies to resolve before the applet source code is run. In this applet, we specify pip3 as our package manager and pysam version 0.15.4 as the dependency to resolve.

Downloading Input

The fields mappings_sorted_bam and mappings_sorted_bai are passed to the main function as parameters for our job. These parameters are dictionary objects with key-value pair {"$dnanexus_link": "<file>-<xxxx>"}. We handle file objects from the platform through DXFile handles. If an index file is not supplied, then a *.bai index will be created.

    print(mappings_sorted_bai)
    print(mappings_sorted_bam)

    mappings_sorted_bam = dxpy.DXFile(mappings_sorted_bam)
    sorted_bam_name = mappings_sorted_bam.name
    dxpy.download_dxfile(mappings_sorted_bam.get_id(),
                         sorted_bam_name)
    ascii_bam_name = unicodedata.normalize(  # Pysam requires ASCII not Unicode string.
        'NFKD', sorted_bam_name).encode('ascii', 'ignore')

    if mappings_sorted_bai is not None:
        mappings_sorted_bai = dxpy.DXFile(mappings_sorted_bai)
        dxpy.download_dxfile(mappings_sorted_bai.get_id(),
                             mappings_sorted_bai.name)
    else:
        pysam.index(ascii_bam_name)

Working with Pysam

Pysam provides several methods that mimic SAMtools commands. In our applet example, we want to focus only on canonical chromosomes. Pysam’s object representation of a BAM file is pysam.AlignmentFile.

    mappings_obj = pysam.AlignmentFile(ascii_bam_name, "rb")
    regions = get_chr(mappings_obj, canonical_chr)

The helper function get_chr

def get_chr(bam_alignment, canonical=False):
    """Helper function to return canonical chromosomes from SAM/BAM header

    Arguments:
        bam_alignment (pysam.AlignmentFile): SAM/BAM pysam object
        canonical (boolean): Return only canonical chromosomes
    Returns:
        regions (list[str]): Region strings
    """
    regions = []
    headers = bam_alignment.header
    seq_dict = headers['SQ']

    if canonical:
        re_canonical_chr = re.compile(r'^chr[0-9XYM]+$|^[0-9XYM]')
        for seq_elem in seq_dict:
            if re_canonical_chr.match(seq_elem['SN']):
                regions.append(seq_elem['SN'])
    else:
        regions = [''] * len(seq_dict)
        for i, seq_elem in enumerate(seq_dict):
            regions[i] = seq_elem['SN']

    return regions

Once we establish a list of canonical chromosomes, we can iterate over them and perform Pysam’s version of samtools view -c, pysam.AlignmentFile.count.

    total_count = 0
    count_filename = "{bam_prefix}_counts.txt".format(
        bam_prefix=ascii_bam_name[:-4])

    with open(count_filename, "w") as f:
        for region in regions:
            temp_count = mappings_obj.count(region=region)
            f.write("{region_name}: {counts}\n".format(
                region_name=region, counts=temp_count))
            total_count += temp_count

        f.write("Total reads: {sum_counts}".format(sum_counts=total_count))

Uploading Outputs

Our summarized counts are returned as the job output. We use the dx-toolkit python SDK’s dxpy.upload_local_file function to upload and generate a DXFile corresponding to our tabulated result file.

    counts_txt = dxpy.upload_local_file(count_filename)
    output = {}
    output["counts_txt"] = dxpy.dxlink(counts_txt)

    return output

Python job outputs have to be a dictionary of key-value pairs, with the keys being job output names as defined in the dxapp.json file and the values being the output values for corresponding output classes. For files, the output type is a DXLink. We use the dxpy.dxlink function to generate the appropriate DXLink value.

View full source code on GitHub

Creating the web application

After configuring an app with Dash, we start the server on port 443.

app.run_server(host='0.0.0.0', port=443)

Inside the dxapp.json, you would add "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose this port.

Note that for all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server will keep it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts will not be executed.

The rest of these instructions apply to building any applet with dependencies stored in an asset.

Creating an applet on DNAnexus

Source dx-toolkit and log in, then run dx-app-wizard with default options.

Creating the asset

dash-asset specifies all the packages and versions we need. We take these from the Dash installation guide (https://dash.plot.ly/installation\)

pip install dash==0.39.0  # The core dash backend
pip install dash-html-components==0.14.0  # HTML components
pip install dash-core-components==0.44.0  # Supercharged components
pip install dash-table==3.6.0  # Interactive DataTable component (new!)
pip install dash-daq==0.1.0  # DAQ components (newly open-sourced!)

We put these into dash-asset/dxasset.json:

{
  ...
  "execDepends": [
    {"name": "dash", "version":"0.39.0", "package_manager": "pip"},
        {"name": "dash-html-components", "version":"0.14.0", "package_manager": "pip"},
        {"name": "dash-core-components", "version":"0.44.0", "package_manager": "pip"},
        {"name": "dash-table", "version":"3.6.0", "package_manager": "pip"},
        {"name": "dash-daq", "version":"0.1.0", "package_manager": "pip"}
  ],
    ...
}

Build the asset:

dx build_asset dash-asset

Use the asset from the applet

Add this asset to the applet’s dxapp.json:

"runSpec": {
    ...
    "assetDepends": [
    {
      "id": "record-xxxx
    }
  ]
    ...
}

Build the applet

Now build and run the applet itself:

dx build -f dash-web-app
dx run dash-web-app

You can always use dx ssh job-xxxx to ssh into the worker and inspect what’s going on or experiment with quick changes Then go to that job’s special URL https://job-xxxx.dnanexus.cloud/ and see the result!

Optional local testing

The main code is in dash-web-app/resources/home/dnanexus/my_app.py with a local launcher script called local_test.py in the same folder. This allows us to launch the same core code in the applet locally to quickly iterate. This is optional because you can also do all testing on the platform itself.

Install locally the same libraries listed above.

To launch the web app locally:

cd dash-web-app/resources/home/dnanexus/
python local_test.py

Once it spins up, you can go to that job’s designated URL based on its job ID, https://job-xxxx.dnanexus.cloud/, to see the result.

There are many definitions and approaches to tackling the concept of parallelization and distributing workloads in the cloud (Here’s a particularly helpful Stack Exchange post on the subject). To help make our documentation easier to understand, when discussing concurrent computing paradigms we’ll refer to:

• Parallel: Using multiple threads or logical cores to concurrently process a workload.

• Distributed: Using multiple machines (in our case instances in the cloud) that communicate to concurrently process a workload.

Keep these formal definitions in mind as you read through the tutorials and learn how to compute concurrently on the DNAnexus platform.

View full source code on GitHub

TensorBoard is a web application used to visualize and inspect what is going on inside TensorFlow training. To use TensorBoard, our training script in TensorFlow needs to include code that saves various data to a log directory where TensorBoard can then find the data to display it.

This example uses an example script from the TensorBoard authors. For more guidance on how to use TensorBoard, check out the tensorflow website (external link).

Creating the web application

The applet code runs a training script, which is placed in resources/home/dnanexus/ to make it available in the current working directory of the worker, and then it starts tensorboard on port 443 (HTTPS).

# Start the training script and put it into the background,
# so the next line of code will run immediately
python3 mnist_tensorboard_example.py --log_dir LOGS_FOR_TENSORBOARD &

# Run TensorBoard
tensorboard  --logdir LOGS_FOR_TENSORBOARD --host 0.0.0.0 --port 443

We run the training script in the background to start TensorBoard immediately, which will let us see the results while training is still running. This is particularly important for long-running training scripts.

Note that for all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server will keep it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts will not be executed.

As with all web apps, the dxapp.json must include "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose port 443.

Creating an applet on DNAnexus

Build the asset with the libraries first:

dx build_asset tensorflow_asset

Take the record ID it outputs and add it to the dxapp.json for the applet.

"runSpec": {
    ...
    "assetDepends": [
    {
      "id": "record-xxxx
    }
  ]
    ...
}

Then build the applet

dx build -f tensorboard-web-app
dx run tensorboard-web-app

Once it spins up, you can go to that job’s designated URL based on its job ID, https://job-xxxx.dnanexus.cloud/, to see the result.

The dx command-line client is included in the DNAnexus SDK (dx-toolkit). You can use the dx client to log into the Platform; to upload, browse, and organize data; and to launch analyses.

All the projects and data referenced in this Quickstart are publicly available, so you can follow along step-by-step.

Getting Help

As you work, you can use this index of dx commands as a reference.

At the command line, you can also enter dx helpto see a list of commands, broken down by category. To see a list of commands from a particular category, enter dx help <category>.

To learn what a particular command does, enter dx help <command>, dx <command> -h, or dx <command> -help For example, enter dx help lsto learn about the command dx ls:

$ dx help ls
usage: dx ls [-h] [--color {off,on,auto}] [--delimiter [DELIMITER]]
[--env-help] [--brief | --summary | --verbose] [-a] [-l] [--obj]
[--folders] [--full]
[path]

List folders and/or objects in a folder
... # output truncated for brevity

Before You Begin

To use the command-line interface (CLI), make sure you've installed the DNAnexus Software Development Kit (SDK) available here.

Upgrading the SDK

To update your version of the command-line tool, you can run the command dx upgrade.

Step 1: Log In

The first thing you'll need to do is to log in. If you haven't created a DNAnexus account yet, visit the website and sign up. User signup is not supported on the command line.

$ dx login
Acquiring credentials from https://auth.dnanexus.com
Username: <your username>
Password: <your password>

No projects to choose from.  You can create one with the command "dx new project".  To pick from projects for which you only have VIEW permissions, use "dx select --level VIEW" or "dx select --public".

Your authentication token and your current project settings have now been saved in a local configuration file, and you're ready to start accessing your project.

Step 2: Explore

Public Projects

Let's look inside some of the public projects that have already been set up. From the command line, enter the command:

$ dx select --public --name "Reference Genome Files*"

By running the dx select command and picking a project, you've now done the command-line equivalent of going to the project page for Reference Genome Files: AWS US (East) (platform login required to access this link) on the website. This is a DNAnexus-sponsored project containing popular genomes for you to use when running analyses with your own data.

For more information about the dx select command, please see the Changing Your Current Project page.

Now you can list all of the data in the top-level directory of the project you've just selected by running the command dx ls. You can also see the contents of a folder by running the command dx ls <folder_name>.

$ dx ls
C. Elegans - Ce10/
D. melanogaster - Dm3/
H. Sapiens - GRCh37 - b37 (1000 Genomes Phase I)/
H. Sapiens - GRCh37 - hs37d5 (1000 Genomes Phase II)/
H. Sapiens - GRCh38/
H. Sapiens - hg19 (Ion Torrent)/
H. Sapiens - hg19 (UCSC)/
M. musculus - mm10/
M. musculus - mm9/
$ dx ls "C. Elegans - Ce10/"
ce10.bt2-index.tar.gz
ce10.bwa-index.tar.gz
... # output truncated for brevity

You can avoid typing out the full name of the folder by typing in dx ls C and then pressing <TAB>. The folder name will auto-complete from there.

You don't have to be in a project to inspect its contents. You can also look into another project, and a folder within the project, by giving the project name or ID, followed by a colon (:) and the folder path. Here, we list the contents of the publicly available project "Demo Data" using both its name and ID.

$ dx ls "Demo Data:/SRR100022/"
SRR100022_1.filt.fastq.gz
SRR100022_2.filt.fastq.gz
$ dx ls -l "project-BQbJpBj0bvygyQxgQ1800Jkk:/SRR100022/"
Project: Demo Data (project-BQbJpBj0bvygyQxgQ1800Jkk)
Folder : /SRR100022
State   Last modified       Size     Name (ID)
... # output truncated for brevity

As shown above, you can use the -l flag in conjunction with dx ls to list more details about files, such as the time a file was last modified, its size (if applicable), and its full DNAnexus ID.

Describing DNAnexus Objects

You can use the dx describe command to learn more about files and other objects on the platform. Given a DNAnexus object ID or name, dx describe will return detailed information about the object in question. dx describe will only return results for data objects to which you have access.

Besides describing data and projects (examples for which are shown below), you can also describe apps, jobs, and users.

Describing a File

Below, we describe the reference genome file for C. elegans located in the "Reference Genome Files: AWS US (East)" project that we've been using (which should be accessible from other regions as well). Note that you need to add a colon (:) after the project name, here that would be Reference Genome Files\: AWS US (East): .

$ dx describe "Reference Genome Files\: AWS US (East):/C. Elegans - Ce10/ce10.fasta.gz"
Result 1:
ID                  file-BQbY9Bj015pB7JJVX0vQ7vj5
Class               file
Project             project-BQpp3Y804Y0xbyG4GJPQ01xv
Folder              /C. Elegans - Ce10
Name                ce10.fasta.gz
State               closed
Visibility          visible
Types               -
Properties          Assembly=UCSC ce10,
                    Origin=http://hgdownload.cse.ucsc.edu/goldenPath/ce10/bigZip
                    s/ce10.2bit, Species=Caenorhabditis elegans, Taxonomy
                    ID=6239
Tags                -
Outgoing links      -
Created             Tue Sep 30 18:54:35 2014
Created by          bhannigan
 via the job        job-BQbY8y80KKgP380QVQY000qz
Last modified       Thu Mar  2 12:17:27 2017
Media type          application/x-gzip
archivalState       "live"
Size                29.21 MB, sponsored by DNAnexus

Describing a Project

Below, we describe the publicly available Reference Genome Files project that we've been using.

$ dx describe "Reference Genome Files\: AWS US (East):"
Result 1:
ID                  project-BQpp3Y804Y0xbyG4GJPQ01xv
Class               project
Name                Reference Genome Files: AWS US (East)
Summary             
Billed to           org-dnanexus
Access level        VIEW
Region              aws:us-east-1
Protected           true
Restricted          false
Contains PHI        false
Created             Wed Oct  8 16:42:53 2014
Created by          tnguyen
Last modified       Tue Oct 23 14:15:59 2018
Data usage          0.00 GB
Sponsored data      519.77 GB
Sponsored egress    0.00 GB used of 0.00 GB total
Tags                -
Properties          -
downloadRestricted  false
defaultInstanceType "mem2_hdd2_x2"

Step 3: Create Your Own Project

Now, we'll use the command dx new project to create a new project.

$ dx new project "My First Project"
Created new project called "My First Project"
(project-xxxx)
Switch to new project now? [y/N]: y

The text project-xxxx denotes a placeholder for a unique, immutable project ID. For more information about object IDs, see the Entity IDs page.

You're now ready to start uploading your data and running your own analyses.

Step 4: Upload and Manage Your Data

If you have a sample you would like to analyze, you can use the dx upload command or the Upload Agent if you have installed it. For the purposes of this tutorial, you can also download the file small-celegans-sample.fastq, which represents the first 25000 C. elegans reads from SRR070372. We will use this file again later to run through a sample analysis.

For uploading multiple or large files, we strongly recommend that you use the Upload Agent; it will compress your files and upload them in parallel over multiple HTTP connections and boasts other features such as resumable uploads.

The following command uploads the small-celegans-sample.fastq file into the current directory of the current project. The --wait flag tells dx upload to wait until it has finished uploading the data before returning the prompt and describing the result.

$ dx upload --wait small-celegans-sample.fastq
[===========================================================>] Uploaded (16801690 of 16801690 bytes) 100% small-celegans-sample.fastq
ID              file-xxxx
Class           file
Project         project-xxxx
Folder          /
Name            small-celegans-sample.fastq
State           closed
Visibility      visible
Types           -
Properties      -
Tags            -
Details         {}
Outgoing links  -
Created         Sun Jan  1 09:00:00 2017
Created by      amy
Last modified   Sat Jan  1 09:00:00 2017
Media type      text/plain
Size            16.02 MB

Examining Data

To take a quick look at the first few lines of the file you just uploaded, use the dx head command. By default, it prints the first 10 lines of the given file.

Let's run it on the file we just uploaded and use the -n flag to ask for the first 12 lines (the first 3 reads) of the FASTQ file.

$ dx head -n 12 small-celegans-sample.fastq
@SRR070372.1 FV5358E02GLGSF length=78
TTTTTTTTTTTTTTTTTTTTTTTTTTTNTTTNTTTNTTTNTTTATTTATTTATTTATTATTATATATATATATATATA
+SRR070372.1 FV5358E02GLGSF length=78
...000//////999999<<<=<<666!602!777!922!688:669A9=<=122569AAA?>@BBBBAA?=<96632
@SRR070372.2 FV5358E02FQJUJ length=177
TTTCTTGTAATTTGTTGGAATACGAGAACATCGTCAATAATATATCGTATGAATTGAACCACACGGCACATATTTGAACTTGTTCGTGAAATTTAGCGAACCTGGCAGGACTCGAACCTCCAATCTTCGGATCCGAAGTCCGACGCCCCCGCGTCGGATGCGTTGTTACCACTGCTT
+SRR070372.2 FV5358E02FQJUJ length=177
222@99912088>C<?7779@<GIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIC;6666IIIIIIIIIIII;;;HHIIE>944=>=;22499;CIIIIIIIIIIIIHHHIIIIIIIIIIIIIIIH?;;;?IIEEEEEEEEIIII77777I7EEIIEEHHHHHIIIIIIIIIIIIII
@SRR070372.3 FV5358E02GYL4S length=70
TTGGTATCATTGATATTCATTCTGGAGAACGATGGAACATACAAGAATTGTGTTAAGACCTGCATAAGGG
+SRR070372.3 FV5358E02GYL4S length=70
@@@@@DFFFFFHHHHHHHFBB@FDDBBBB=?::5555BBBBD??@?DFFHHFDDDDFFFDDBBBB<<410

Downloading Data

If you'd like to download a file from the platform, just use the dx download command. This command will use the name of the file for the filename unless you specify your own with the -o/--output flag. In the example below, we download the same C. elegans file that we uploaded previously.

$ dx download small-celegans-sample.fastq
[                                                            ] Downloaded 0 byte
[===========================================================>] Downloaded 16.02 of
[===========================================================>] Completed 16.02 of 16.02 bytes (100%) small-celegans-sample.fastq

About Metadata

Files have different available fields for metadata, such as "properties" (key-value pairs) and "tags".

Step 5: Analyze a Sample

For the next few steps, if you would like to follow along, you will need a C. elegans FASTQ file. We will map the reads against the ce10 genome. If you haven't already, you can download and use the following FASTQ file, which contains the first 25,000 reads from SRR070372: small-celegans-sample.fastq.

The following walkthrough is helpful if you would like to understand what all the commands do and take a look at what apps you're running, but if you're just interested in converting a gzipped FASTQ file to a VCF file via BWA and the FreeBayes variant caller, then you can skip ahead to the Automate It section below, where you can see all the commands necessary for running apps.

Uploading Reads

If you have not yet done so, you can upload a FASTQ file for analysis.

$ dx upload small-celegans-sample.fastq --wait

For more information about using the command dx upload, please see the dx upload page.

Mapping Reads

Next, use the BWA-MEM app (platform login required to access this link) to map the uploaded reads file to a reference genome.

Finding the App Name

If you don't know the command-line name of the app you would like to run, you have two options:

1. You can navigate to its web page from the Apps page (platform login required to access this link) on the platform. The app's page will tell you how to run it from the command line. You can find more information about the app we're running on the BWA-MEM FASTQ Read Mapper page (platform login required to access this link).

2. Alternatively, you can search for apps from the command line by running the command dx find apps. You will find the name of the app that you can use on the command line in the parentheses (underlined below).

$ dx find apps
...
x BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper), v1.4.0
...

Installing and Running the App

Now install the app using dx install and check that it has been installed. While you do not always need to install an app to run it, you may find it useful as a bookmarking tool.

$ dx install bwa_mem_fastq_read_mapper
Installed the bwa_mem_fastq_read_mapper app
$ dx find apps --installed
BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper), v1.4.0

We can now run the app using dx run. We will run it without any arguments; it will then prompt us for required and then optional arguments. Note that the reference file genomeindex_targz for the C. elegans sample we are using is in a .tar.gz format and can be found in the Reference Genome folder of the region your project is in.

$ dx run bwa_mem_fastq_read_mapper
Entering interactive mode for input selection.

Input:   Reads (reads_fastqgz)
Class:   file
Enter file ID or path (<TAB> twice for compatible files in current directory,'?' for help)
reads_fastqgz[0]: <small-celegans-sample.fastq.gz>

Input:   BWA reference genome index (genomeindex_targz)
Class:   file

Suggestions:
project-BQpp3Y804Y0xbyG4GJPQ01xv://file-\* (DNAnexus Reference Genomes)
Enter file ID or path (<TAB> twice for compatible files in current
directory,'?' for more options)
genomeindex_targz: <"Reference Genome Files\: <REGION_OF_PROJECT>:/C. Elegans - Ce10/ce10.bwa-index.tar.gz">

Select an optional parameter to set by its # (^D or <ENTER> to finish):

[0] Reads (right mates) (reads2_fastqgz)
[1] Add read group information to the mappings (required by downstream GATK)? (add_read_group) [default=true]
[2] Read group id (read_group_id) [default={"$dnanexus_link": {"input": "reads_fastqgz", "metadata": "name"}}]
[3] Read group platform (read_group_platform) [default="ILLUMINA"]
[4] Read group platform unit (read_group_platform_unit) [default="None"]
[5] Read group library (read_group_library) [default="1"]
[6] Read group sample (read_group_sample) [default="1"]
[7] Output all alignments for single/unpaired reads? (all_alignments)
[8] Mark shorter split hits as secondary? (mark_as_secondary) [default=true]
[9] Advanced command line options (advanced_options)

Optional param #: <ENTER>

Using input JSON:
{
    "reads_fastqgz": {
        "$dnanexus_link": {
            "project": "project-B3X8bjBqqBk1y7bVPkvQ0001",
            "id": "file-B3P6v02KZbFFkQ2xj0JQ005Y"
        }

"genomeindex_targz": {
        "$dnanexus_link": {
            "project": "project-xxxx(project ID for the reference genome in your region)",
            "id": "file-BQbYJpQ09j3x9Fj30kf003JG"
        }
    }
}

Confirm running the applet/app with this input [Y/n]: <ENTER>
Calling app-BP2xVx80fVy0z92VYVXQ009j with output destination
     project-xxxx:/

Job ID: job-xxxx

Monitoring Your Job

You can use the command dx watch to monitor jobs. The command will print out the log file of the job, including the STDOUT, STDERR, and INFO printouts.

You can also use the command dx describe job-xxxx to learn more about your job. If you don't know the job's ID, you can use the command dx find jobs to list all the jobs run in the current project, along with the user who ran them, their status, and when they began.

$ dx find jobs
* BWA-MEM FASTQ Read Mapper (bwa_mem_fastq_read_mapper:main)(done) job-xxxx
user-amy 20xx-xx-xx 0x:00:00 (runtime 0:00:xx)
$ dx describe job-xxxx
...

There are also additional options that you can use to restrict your search of previous jobs, such as by their names or when they were run.

Terminating Your Job

If for some reason you need to terminate your job before it completes, use the command dx terminate.

After Your Job Finishes

You should now see two new files in your project: the mapped reads in a BAM file, and an index of that BAM file with a .bai extension. You can refer to the output file by name or by the job that produced it using the syntax job-xxxx:<output field>. Try it yourself with the job ID you got from calling the BWA-MEM app!

$ dx ls
small-celegans-sample.bam
small-celegans-sample.bam.bai
small-celegans-sample.fastq
$ dx describe small-celegans-sample.bam
...
$ dx describe job-xxxx:sorted_bam
...

Variant Calling

You can use the FreeBayes Variant Caller app (platform login required to access this link) to call variants on your BAM file.

This time, we won't rely on the interactive mode to enter our inputs. Instead, we will provide them directly. But first, let's look up the app's spec so we know what the inputs are called. For this, let's run the command dx run freebayes -h.

$ dx run freebayes -h
usage: dx run freebayes [-iINPUT_NAME=VALUE ...]

App: FreeBayes Variant Caller

Calls variants (SNPs, indels, and other events) using FreeBayes

See the app page for more information:
https://platform.dnanexus.com/app/freebayes

Inputs:
    Sorted mappings: -isorted_bams=(file) [-isorted_bams=... [...]]
        One or more coordinate-sorted BAM files containing mappings to call
        variants for.

    Genome: -igenome_fastagz=(file)
        A file, in gzipped FASTA format, with the reference genome that the
        reads were mapped against.
...

Optional inputs are shown using square brackets ([]) around the command-line syntax for each input. You'll notice that there are two required inputs that must be specified:

1. Sorted mappings (sorted_bams): A list of files with a .bam extension.

2. Genome (genome_fastagz): A reference genome in FASTA format that has been gzipped.

Running the App with a One-Liner Using a Job-Based Object Reference

It is sometimes more convenient to run apps using a single one-line command. You can do this by specifying all the necessary inputs either via the command line or in a prepared file. We will use the -i flag to specify inputs as suggested by the output of dx run freebayes ‑h:

• sorted_bams: The output of the previous BWA step (see the Map Reads section for more information).

• genome_fastagz: The ce10 genome in the Reference Genomes project.

To specify new job input using the output of a previous job, we'll use a [job-based object reference](/Developer-Tutorials/Sample-Code?bash#Use-job-based-object-references-(JBORs)) via the job-xxxx:<output field> syntax we used earlier.

Replace the job ID below with that generated by the BWA app you ran earlier. The -y flag skips the input confirmation.

$ dx run freebayes -y \ 
 -igenome_fastagz=Reference\ Genome\ Files:/C.\ Elegans\ -\ Ce10/ce10.fasta.gz \ 
 -isorted_bams=job-xxxx:sorted_bam

Using input JSON:
{
  "genome_fastagz": {
    "$dnanexus_link": {
      "project": "project-xxxx",
      "id": "file-xxxx"
    }
  },
  "sorted_bams": {
    "field": "sorted_bam", 
    "job": "job-xxxx"
  } 
}

Calling app-BFG5k2009PxyvYXBBJY00BK1 with output destination
project-xxxx:/

Job ID: job-xxxx

Automatically Running a Command After a Job Finishes

You can use the command dx wait to wait for a job to finish. If we run the following command right after running the Freebayes app, it will show you the recent jobs only after the job has finished, as shown in the example below.

$ dx wait job-xxxx && dx find jobs
Waiting for job-xxxx to finish running...
Done
* FreeBayes Variant Caller (done) job-xxxx
user-amy 2017-01-01 09:00:00 (runtime 0:05:24)
...

Congratulations! You have now called variants on a reads sample, and you did it all on the command line. Now let's look at how you can automate this process.

Automation

The beauty of the CLI is the ability to automate processes. In fact, we can automate everything we just did. The following script assumes that you've already logged in and is hardcoded to use the ce10 genome and takes in a local gzipped FASTQ file as its command-line argument.

#!/usr/bin/env bash
# Usage: <script_name.sh> local_fastq_filename.fastq.gz

reference="Reference Genome Files\: AWS US (East):/C. Elegans - Ce10/ce10.fasta.gz"
bwa_indexed_reference="Reference Genome Files\: AWS US (East):/C. Elegans - Ce10/ce10.bwa-index.tar.gz"
local_reads_file="$1"

reads_file_id=$(dx upload "$local_reads_file" --brief)
bwa_job=$(dx run bwa_mem_fastq_read_mapper -ireads_fastqgzs=$reads_file_id -igenomeindex_targz="$bwa_indexed_reference" -y --brief)
freebayes_job=$(dx run freebayes -isorted_bams=$bwa_job:sorted_bam -igenome_fastagz="$reference" -y --brief)

dx wait $freebayes_job

dx download $freebayes_job:variants_vcfgz -o "$local_reads_file".vcf.gz
gunzip "$local_reads_file".vcf.gz

Learn More

You're now ready to start scripting using dx. As shown in some of the examples above, the --brief flag can come in handy for scripting. A list of all dx commands and flags is on the Index of dx Commands page.

For more detailed information about running apps and applets from the command line, see the Running Apps and Applets page.

For a comprehensive guide to the DNAnexus SDK, see the SDK documentation.

Want to start writing your own apps? Check out the Developer Portal for some useful tutorials.

Learn More

See these Key Concepts pages to learn more about how the DNAnexus Platform works, and how to get the most from it:

• Projects

• Organizations

• Apps and Workflows

Get up and running quickly using the Platform via both its user interface (UI) and its command-line interface (CLI):

• User Interface Quickstart

• Command Line Quickstart

Learn the basics of developing for the Platform:

• Developer Quickstart

• Developer Tutorials

Uploading and Sharing Data

Running a Single App

Creating and Running a Workflow

Monitoring Jobs and Viewing Results

Visualizing Data

Learn More

See these Key Concepts pages for more in-depth treatments of topics that are covered briefly here:

• Projects

• Organizations

• Apps and Workflows

For a step-by-step written tutorial to using the Platform via its UI, see this User Interface Quickstart.

For a step-by-step written tutorial to using the Platform via its CLI, see this Command Line Quickstart.

For a more in-depth video intro to the Platform, watch this DNAnexus Platform Essentials video.

Additional Tutorials

As a developer, you may be interested in the following:

• dxCompiler

• JupyterLab

• Spark on JupyterLab

As a bioinformatician, you may be interested in our SAIGE GWAS walkthrough, and other content grouped in our Science Corner.

Using the DNAnexus Platform

Get to know features you’ll use every day, in a series of short, task-oriented tutorials.

Learn to access and use the Platform via both its command-line interface and its user interface.

Learn to manage data, users, and work on the Platform, via its API. Create and share reusable pipelines, applications for analyzing data, custom viewers, and workflows.

This section is targeted towards organizational leads who have the permission to enable others to use DNAnexus for scientific purposes. Operations include managing organization permissions, billing, and authentication to the platform.

Download, install, and get started using the DNAnexus Platform SDK, the DNAnexus upload and download agents, and dxCompiler.

Get details on new features, changes, and bug fixes for each Platform and toolkit release.

Finding the Right App or Workflow

The Tools Library provides a list of available apps and workflows. To see this list, select Tools Library from the Tools entry in the main Platform menu.

To find the tool you're looking for in the Tools Library, you can use search filters. Filtering enables you to find tools with a specific name, in a specific category, or of a specific type:

To see what inputs a tool requires, and what outputs it generates, select that tool's row in the list. The row will be highlighted in blue; the tool's inputs and outputs will be displayed in a pane to the right of the list:

To make sure you can find a tool later, "pin" it to the top of the list. Click the "..." icon at the far right end of the row showing the tool's name and key details about it. Then click Add Pin:

To learn more about a tool, click on its name in the list. The tool's detail page will open, showing a wide range of info, including guidance in how to use it, version history, pricing, and more:

Running Apps and Workflows

Launching a Tool

Launching from the Tools Library

You can quickly launch the latest version of any given tool from the Tools Library page. Or, navigate into the Details page of the tool and click the Run button:

Launching from a Project

From within a project, navigate to the Manage pane, then click the Start Analysis button.

A dialogue window will open, showing a list of tools. These will include the same tools as shown in the Tools Library, as well as workflows and applets specifically available in the current project. Select the tool you want to run, then click Run Selected:

Workflows and applets can be launched directly from where they reside within a project. Select the workflow or applet in their folder location, and click Run.

Launch Configuration

Confirm details of the tool you are about to run. Note that selection of a project location is required for any tool to be run. You will need at minimum Contributor access level to the project.

Configure Inputs and Outputs

The tool may require specific inputs to be filled in before starting the run. You can quickly identify the required inputs by looking for the highlighted areas that are marked Inputs Required on the page.

You can access help information about each input or output by inspecting the label of each item. If a detailed README is provided for the executable, you can click the View Documentation icon to open the app or workflow info pane.

To configure instance type settings for a given tool or stage, click the Instance Type icon located on the top-right corner of the stage.

To configure output location and view info regarding output items, go to the Outputs tab under each stage. For workflows, output location can be specified separately for each stage.

The I/O graph provides an overview of the input/output structure of the tool. The graph is available for any tool and can be accessed via the Actions/Workflow Actions menu.

Once all required inputs have been configured, the page will indicate that the run is ready to start. Click on Start Analysis to proceed to the final step.

Configure Runtime Settings

As the last step before launching the tool, you can review and confirm various runtime settings, including execution name, output location, priority, job rank, spending limit, etc. You can also review and modify instance type settings before starting the run.

Once you have confirmed final details, click Launch Analysis to start the run.

Batch Run

Batch run allows users to run the same app or workflow multiple times, with specific inputs varying between runs.

Specify Batch Inputs

To enable batch run, start from any input that you wish to specify for batch run, and open its I/O Options menu on the right hand side. From the list of options available, select Enable Batch Run.

Input fields with batch run enabled will be highlighted with a Batch label. Click any of the batch enabled input fields to enter the batch run configuration page.

Configure Batch Inputs

The batch run configuration page allows specifying inputs across multiple runs. Interact with each table cell to fill in desired values for any run or field.

Similar to configuration of inputs for non-batch runs, you will need to fill all the required input fields to proceed to next steps. Optional inputs, or required inputs with a predefined default value, can be left empty.

Once all required fields (for both batch inputs and non-batch inputs) have been configured, you can proceed to start the run via the Start Analysis button.

Starting and Monitoring Your Analysis

Once you've finished setting up your tool, start your analysis by clicking the Start Analysis button. Follow these instructions to monitor the job as it runs.

Learn More

Learn in depth about running apps and workflows, leveraging advanced techniques like Smart Reuse.

Learn how to build a simple app.

Learn more about building apps using Bash or Python.

Learn in depth about building and deploying apps, including Spark apps.

Learn in depth about importing, building, and running workflows.

What Is an Org?

An organization (or "org") is a DNAnexus entity used to manage a group of users. Use orgs to group users, projects, and other resources together, in a way that models real-world collaborative structures.

In its simplest form, an org can be thought of as referring to a group of users on the same project. An org can be used efficiently to share projects and data with multiple users - and, if necessary, to revoke access.

Org admins can manage org membership, configure access and projects associated with the org, and oversee billing. All storage and compute costs associated with an org are invoiced to a single billing account designated by the org admin. You can create an org that is associated with a billing account by contacting sales@dnanexus.com.

Orgs are referenced on the DNAnexus Platform by a unique org ID (e.g., org-dnanexus). Org IDs are used when sharing projects with an org in the Platform user interface or when manipulating the org in the CLI.

Org Membership Levels

Users may have one of two membership levels in an org:

• ADMIN

• MEMBER

An ADMIN-level user is granted all possible access in the org and may perform org administrative functions (e.g. adding/removing users or modifying org policies). A MEMBER-level user, on the other hand, is granted only a subset of the possible org accesses in the org and has no administrative power in the org.

Members

A user with MEMBER level can be configured to have a subset of the following org access. These access levels determine which actions each user can perform in an org.

These accesses allow you to have fine-grained control over what members of your orgs can do in the context of your org.

Admins

Org admins are granted all possible access in the org. More specifically, org admins receive the following set of accesses:

In addition to the access listed above, org admins have the following additional abilities:

Viewing Metadata for All Org Projects

Org admins can list and view metadata for all org projects (projects billed to the org) even if the project is not explicitly shared with them. They can also give themselves access to any project billed to the org. For example, when a member creates a new project, Project-P, and bills it to the org, he is the only user with access to Project-P. The org admin will be able to see all projects billed to the org, including Project-P. The org admin can also invite themself to Project-P at any time to get access to objects and jobs in the project.

Becoming a Developer for All Org Apps

Org admins can add themselves as a developer to any app billed to the org. For example, when a member creates a new app, App-A, billed to the org, he is the only developer for App-A; however, any org admin may add themself as a developer at any time.

Examples of Using Orgs

Org Structure Diagram

In the diagram below, there are 3 examples of how organizations can be structured.

ORG-1

The simplest example, ORG-1, is represented by the leftmost circle. In this situation, ORG-1 is a billable org that has 3 members who share one billing account, so all 5 projects created by the members of ORG-1 are billed to that org. There is one admin who manages ORG-1, represented here as user A.

ORG-2 and ORG-3

The second example shows ORG-2 and ORG-3 demonstrating more a complicated organizational setup. Here users are grouped into two different billable orgs, with some users belonging to both orgs and others belonging to only one.

In this case, ORG-2 and ORG-3 bill their work against separate billing accounts. This separation of orgs can represent two different groups in one company working in different departments, each with their own budgets, two different labs that work closely together, or any other scenario in which two collaborators would share work.

ORG-2 has 5 members, 4 projects, and is managed by one org admin (user G). ORG-3 has 5 members and 3 projects, but is managed by 2 admins (users G and I).

In this example, admin G and member H belong to both ORG-2 and ORG-3. They can create new projects billed to either org, depending on the project they're working on. Admin G can manage users and projects in both ORG-2 and ORG-3.

Example 1: Creating an Org for Sharing Data

You can create a non-billable org as an alias for a group of users. For example, you have a group of users who all need access to a shared dataset. You can make an org which represents all the users who need access to the dataset, for example an org named org-dataset_access, and share all the projects and apps related to the dataset with that org. All members of the org will have at least VIEW "shared project access" and "shared app access" so that they will all be given permission to view the dataset. If a member no longer needs access to the dataset, they can be removed from the org, and will then no longer have access to any projects or apps shared with org-dataset_access.

Example 2: Only Admins can Create Projects

You can request sales@dnanexus.com to create a billable org where only one member, the org admin, can create new org projects. All other org members will not be granted the "billable activities access", and so cannot create new org projects. The org admin can then assign each org member a "shared projects access" (VIEW, UPLOAD, CONTRIBUTE, ADMINISTER) and share every org project with the org with ADMINISTER access. The members' permissions to the projects will be restricted by their respective "shared project access."

For example, in a given group, bioinformaticians can be given CONTRIBUTE access to the projects shared with the entire org, so they can run analyses and produce new data in any of the org projects. However, the sequencing center technicians only need UPLOAD permissions to add new data to the projects. Analysts in the group will only be given VIEW access to projects shared with the org. When you need to add a new member to your group and give them access to the projects shared with the org, you simply need to add them to the org as a new member and assign them the appropriate permission levels.

This membership structure allows the org admin to control the number of projects billed to the org. The org admin can also quickly share new projects with their org and revoke permissions from users who have been removed from the org.

Example 3: Shared Billing Account

You can request sales@dnanexus.com to create a billable org where users work independently and bill all of their activities to the org billing account (as specified by the org admin). All org members are granted "billable activities access." The org members also need to share common resources (e.g. incoming samples or reference datasets).

In this case, all members should be granted the "shared apps access" and assigned VIEW as their "shared projects access." The reference datasets that need to be shared with the org are stored in an "Org Resources" project that is shared with the org, which is granted VIEW access. The org can also have best-practice executables built as apps on the DNAnexus system.

The apps can be shared with the org so all members of the org have access to these (potentially proprietary) executables. If any user leaves your company or institution, their access to reference datasets and executables is revoked simply by removing them from the org.

Other Cases

In general, it is possible to apply many different schemas to orgs as they were designed for many different real-life collaborative structures. If you have a type of collaboration you would like to support, contact support@dnanexus.com for more information about how orgs can work for you.

Managing Your Orgs

If you are an admin of an org, you can access the org admin tools from the Org Admin link in the header of the DNAnexus Platform. From here, you can quickly navigate to the list of orgs you administer via All Orgs, or to a specific org.

The Organizations list shows you the list of all orgs to which you have admin access. On this page, you can quickly see all of your orgs, the org IDs, their Project Transfer setting, and the Member List Visibility setting.

Within an org, the Settings tab allows you to view and edit basic information, billing, and policies for your org.

Viewing and Updating Org Information

You can find the org overview on the Settings tab. From here, you can:

• View and edit the organization name (this is how the org is referred to in the Platform user interface and in email notifications).

• View the organization ID, the unique ID used to reference a particular org on the CLI (e.g. org-demo_org).

• View the number of org members, org projects, and org apps.

• View the list of organization admins.

Managing Org Members

Within an org page, the Members tab allows you to view all the members of the org, invite new members, remove existing members, and update existing members' permission levels.

From the Members tab, you can quickly see the names and access levels for all org members. For more information about org membership, see the organization member guide.

Inviting a New Member

To add existing DNAnexus user to your org, you can use the + Invite New Member button from the org's Members tab. This opens a screen where you can enter the user's username (e.g., smithj) or user-ID (e.g., user-smithj). Then you can configure the user's access level in the org.

If you add a member to the org with billable activities access set to billing allowed, they will have the ability to create new projects billed to the org.

However, adding the member will not change their default billing account. If the user wishes to use the org as their default billing account, they will have to set their own default billing account.

Additionally, if the member has any pre-existing projects that are not billed to the org, the user will need to transfer the project to an org if they wish to have the project billed to the org.

The user will receive an email notification informing them that they have been added to the organization.

Creating New DNAnexus Accounts

Org admins have the ability to create new DNAnexus accounts on behalf of the org, provided the org is covered by a license that enables account provisioning. The user will then receive an email with instructions to activate their account and set their password.

If this feature has already been turned on for an org you administer, you will see an option to Create New User when you go to invite a new member.

Here you can specify a username (e.g. alice or smithj), the new user's name, and their email address. The system will automatically create a new user account for the given email address and add them as a member in the org.

If you create a new user and set their Billable Activities Access to Billing Allowed, we recommend that you set the org as the user's default billing account. This option is available as a checkbox under the Billable Activities Access dropdown.

Editing Member Access

From the org Members tab, you can edit the permissions for one or multiple members of the org. The option to Edit Access appears when you have one or more org members selected in the table.

When you edit multiple members, you have the option of changing only one access while leaving the rest alone.

Removing Members

From the org Members tab, you can remove one or more members from the org. The option to Remove appears when you have one or more org members selected on the Members tab.

Removing a member revokes the user's access to all projects and apps billed to or shared with the org.

Org Projects

In the org's Projects tab you to see the list of all projects billed to the org. This list includes all projects in which you have VIEW and above permissions as well as projects that are billed to the org in which you do not have permissions (not a Member of).

You can view all project metadata (e.g. the list of members, data usage, creation date), as well as some other optional columns (e.g. project creator). To enable the optional columns, select the column from the dropdown menu to the right of the column names.

Granting Admin Access to Org Projects

In addition to viewing the list of projects, org admins can give themselves access to any project billed to the org. If you select a project in which you are not a member, you will still be able to navigate into the project's settings page. On the project settings page, you can click a button to grant yourself ADMINISTER permissions to the project.

You can also grant yourself ADMINISTER permissions if you are currently a member of a project billed to your org but you only have VIEW, CONTRIBUTE, or UPLOAD permissions.

Org Billing

Accessing Org Billing Information

To access your org's billing information:

1. From the global menu shown at the top of the screen, click org admin and then click the name of the org you want to view.

2. Click Billing from the org details screen to view billing information.

Setting Up or Updating Billing Information for an Org

To set up or update the billing information for an org you administer, contact billing@dnanexus.com.

Setting a Spending Limit

If you are an org admin, you can set or modify a spending limit for the org's usage charges. To do this:

1. Click on Org Admin from the global menu bar and then click on the name of the org for which you'd like to set or modify a spending limit.

2. Once on your org page, click on the billing tab.

3. Click the Set / Update Spending Limit link in the Funds Left box to contact DNAnexus Support, requesting a spending limit change. Doing this only submits your request. DNAnexus Support may follow up with you via email with questions about the change, before approving the request.

Viewing Estimated Charges

In the Funds Left section shows how much is left of the org's spending limit. If your org does not have a spending limit, your org is unlimited, which shows up as “N/A.”

Monitoring Spending and Usage

The Per-Project Usage Report and Root Execution Stats Report are monthly reports that provide a wide range of detail on charges incurred by org members. See this documentation for more information.

Org Policies

Org admins can also set configurable policies for the org. Org policies dictate many different behaviors when the org interacts with other entities. The following policies exist:

DNAnexus recommends, as a starting point, to restrict the "membership list visibility policy" to ADMIN and "project transfer policy" to ADMIN. This ensures that only the org admin is allowed to see the list of members and their access within the org and that org projects always remain under control of the org.

You can update org policies for your org in the Policies and Administration section of the org Settings tab. Here, you can both change the membership list visibility and restrict project transfer policies for the org and contact DNAnexus support to enable PHI data policies for org projects.

Glossary of Org Terms

• Billable activities access is an access level that can be granted to org members. If allowed, the org member can create new projects and apps billed to the org, download data (incurring data egress charges against the org), and set their own default billing account to that of the org.

• Billable org is an org that has confirmed billing information or a non-negative spending limit remaining. Users with billable activities access in a billable org are allowed to create new projects billed to the org. See the definition of a non-billable org for an org that is used for sharing.

• Billed to an org (app context) sets the billing account of an app to an org. Apps require storage for their resources and assets, and the billing account of the app are billed for that storage. The billing account of an app does not pay for invocations of the app unless the app is run in a project billed to the org.

• Billed to an org (project context) sets the billing account of a project to an org. The org is invoiced the storage for all data stored in the project as well as compute charges for all jobs and analyses run in the project.

• Membership level describes one of two membership levels available to users in an org, ADMIN or MEMBER. Note that ADMINISTER is a type of access level.

• Membership list visibility policy dictates the minimum org membership level required to view the list of org members, their membership level, and access within the org.

• Non-billable org describes an org only used as an alias for a group of users. Non-billable orgs do not have billing information and will not have any org projects or org apps. Any user can share a project with a non-billable org.

• Org access is granted to a user to determine which actions the user can perform in an org.

• Org admin describes administrators of an org who can manage org membership, configure access and projects associated with the org, and oversee billing.

• Org app is an app billed to an org.

• Org ID is the unique ID used to reference a particular org on the DNAnexus Platform (e.g. org-dnanexus).

• Org member is a DNAnexus user associated with an org. Org members can have variable membership levels in an org which define their role in the org. Admins are a type of org member as well.

• Org policy is a configurable policy for the org. Org policies dictate many different behaviors when the org interacts with other entities.

• Org project describes a project billed to an org.

• Org (or "organization") is a DNAnexus entity that is used to associate a group of users. Orgs are referenced on the DNAnexus Platform by a unique org ID.

• Project transfer policy dictates the minimum org membership level allowed to change the billing account of an org project.

• Share with an org means to give the members of an org access to a project or app via giving the org access to the project or adding the org as an "authorized user" of an app.

• Shared apps access is an org access level that can be granted to org members. If allowed, the org member can view and run apps in which the org has been added as an "authorized user."

• Shared projects access is an org access level that can be granted to org members: the maximum access level a user can have in projects shared with an org.

Learn More

Learn in depth about setting up and managing orgs as an administrator.

Learn about what you can do as an org member.

Learn about creating and managing orgs as a developer, via the DNAnexus API.

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json file’s runSpec.execDepends.

For additional information, see execDepends.

Entry Points

Distributed bash-interpreter apps use bash functions to declare entry points. This app has the following entry points specified as bash functions:

• main

• count_func

• sum_reads

Entry points are executed on a new worker with its own system requirements. The instance type can be set in the dxapp.json file’s runSpec.systemRequirements:

main

The main function slices the initial *.bam file and generates an index *.bai if needed. The input *.bam is the sliced into smaller *.bam files containing only reads from canonical chromosomes. First, the main function downloads the BAM file and gets the headers.

Outputs from the count_func entry points are referenced as Job Based Object References (JBOR) and used as inputs for the sum_reads entry point.

The output of the sum_reads entry point is used as the output of the main entry point via JBOR reference using the command dx-jobutil-add-output.

count_func

This entry point downloads and runs the command samtools view -c on the sliced *.bam. The generated counts_txt output file is uploaded as the entry point’s job output via the command dx-jobutil-add-output.

sum_reads

The main entry point triggers this sub job, providing the output of count_func as an input. This entry point gathers all the files generated by the count_func jobs and sums them.

This function returns read_sum_file as the entry point output.