Using the DNAnexus Platform

Get to know commonly used features in a series of short, task-oriented tutorials.

Learn to access and use the Platform via both its command-line interface and its user interface.

Learn to manage data, users, and work on the Platform, via its API. Create and share reusable pipelines, applications for analyzing data, custom viewers, and workflows.

This section is targeted towards organizational leads who have the permission to enable others to use DNAnexus for scientific purposes. Operations include managing organization permissions, billing, and authentication to the platform.

Download, install, and get started using the DNAnexus Platform SDK, the DNAnexus upload and download agents, and dxCompiler.

Get details on new features, changes, and bug fixes for each Platform and toolkit release.

Learn More

See these Key Concepts pages to learn more about how the DNAnexus Platform works, and how to get the most from it:

To use the CLI, you need to .

If you're not familiar with the dx client, check the .

This section provides detailed instructions on using the dx client to perform such common actions as logging in, selecting projects, listing, copying, moving, and deleting objects, and launching and monitoring jobs. Details on using the UI are included throughout, as applicable.

A project is a collaborative workspace on the DNAnexus Platform where you can store objects such as files, applets, and workflows. Within projects, you can run apps and workflows. You can also share a project with other users by giving them access to it. Read about projects in the Key Concepts section.

Many definitions and approaches exist for tackling the concept of parallelization and distributing workloads in the cloud (Here's a particularly helpful Stack Exchange post on the subject). To help make the documentation easier to understand, when discussing concurrent computing paradigms this guide refers to:

• Parallel: Using multiple threads or logical cores to concurrently process a workload.

• Distributed: Using multiple machines (in this case instances in the cloud) that communicate to concurrently process a workload.

Keep these formal definitions in mind as you read through the tutorials and learn how to compute concurrently on the DNAnexus Platform.

Uploading and Sharing Data

Running a Single App

Creating and Running a Workflow

Monitoring Jobs and Viewing Results

Visualizing Data

Learn More

See these Key Concepts pages for more in-depth treatments of topics that are covered briefly here:

For a step-by-step written tutorial to using the Platform via its UI, see .

For a step-by-step written tutorial to using the Platform via its CLI, see .

For a more in-depth video intro to the Platform, watch the .

Additional Tutorials

As a developer, you may be interested in the following:

As a bioinformatician, see the walkthrough and other content in the .

View full source code on GitHub

TensorBoard is a web application used to visualize and inspect what is going on inside TensorFlow training. To use TensorBoard, the training script in TensorFlow needs to include code that saves specific data to a log directory where TensorBoard can then find the data to display it.

This example uses an example script from the TensorBoard authors. For more guidance on how to use TensorBoard, check out the TensorFlow website (external link).

Creating the web application

The applet code runs a training script, which is placed in resources/home/dnanexus/ to make it available in the current working directory of the worker, and then it starts TensorBoard on port 443 (HTTPS).

The training script runs in the background to start TensorBoard immediately, which allows you to see the results while training is still running. This is particularly important for long-running training scripts.

For all web apps, if everything is running smoothly and no errors are encountered (the ideal case), the line of code that starts the server keeps it running forever. The applet stops only when it is terminated. This also means that any lines of code after the server starts are not executed.

As with all web apps, the dxapp.json must include "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose port 443.

Creating an applet on DNAnexus

Build the asset with the libraries first:

Take the record ID it outputs and add it to the dxapp.json for the applet.

Then build the applet

Once it spins up, you can go to that job's designated URL based on its job ID, https://job-xxxx.dnanexus.cloud, to see the result.

View full source code on GitHub

Precompiling a Binary

In this applet, the SAMtools binary was precompiled on an Ubuntu machine. A user can do this compilation on an Ubuntu machine of their own, or they can use the Cloud Workstation app to build and compile a binary. On the Cloud Workstation, the user can download the SAMtools source code and compile it in the worker environment, ensuring that the binary runs on future workers.

See Cloud Workstationin the App library for more information.

Resources Directory

The SAMtools precompiled binary is placed in the <applet dir>/resources/ directory. Any files found in the resources/ directory are packaged, uploaded to the Platform, and then extracted in the root directory \ of the worker. In this case, the resources/ dir is structured as follows:

When this applet is run on a worker, the resources/ directory is placed in the worker's root directory /:

The SAMtools command is available because the respective binary is visible from the default $PATH variable. The directory /usr/bin/ is part of $PATH, so the script can reference the samtools command directly:

Why Run an Older Version of DXJupyterLab?

The primary reason to run an older version of DXJupyterLab is to access snapshots containing tools that cannot be run in the current version's execution environment.

Launching an Older Version via the User Interface (UI)

1. From the main Platform menu, select Tools, then Tools Library.

2. Find and select, from the list of tools, either DXJupyterLab with Python, R, Stata, ML, Image Processing or DXJupyterLab with Spark Cluster.

3. From the tool detail page, click on the Versions tab.

Launching an Older Version via the Command-Line Interface (CLI)

1. Select the project in which you want to run DXJupyterLab.

2. Launch the version of DXJupyterLab you want to run, substituting the version number for x.y.z in the following commands:

   • For DXJupyterLab without Spark cluster capability, run the command dx run app-dxjupyterlab/x.y.z --priority high.

Accessing DXJupyterLab

After launching DXJupyterLab, access the DXJupyterLab environment using your browser. To do this:

1. Get the job ID for the job created when you launched DXJupyterLab. See the for details on how to get the job ID, via either the UI or the CLI.

2. Open the URL https://job-xxxx.dnanexus.cloud, substituting the job's ID for job-xxxx.

3. You may see an error message "502 Bad Gateway" if DXJupyterLab is not yet accessible. If this happens, wait a few minutes, then try again.

Spark Applications

The Spark application is an extension of the current app(let) framework. App(let)s have a specification for their VM (instance type, OS, packages). This has been extended to allow for an additional optional cluster specification with type=dxspark.

• Calling /app(let)-xxxx/run for Spark apps creates a Spark cluster (+ master VM).

• The master VM (where the app shell code runs) acts as the driver node for Spark.

• Code in the master VM leverages the Spark infrastructure.

• Job mechanisms (monitoring, termination, and management) are the same for Spark apps as for any other regular app(let)s on the Platform.

Spark apps can be launched over a distributed Spark cluster.

On the DNAnexus Platform, the following terms are used when discussing apps and workflows:

• Execution: An analysis or job.

  • Root execution: The initial analysis or job that's created when a user makes an API call to run a workflow, app, or applet. Analyses and jobs created from a job via /executable-xxxx/run API call with detach flag set to true are also root executions.

  • Execution tree: The set of all jobs and/or analyses that are created because of running a root execution.

• Analysis: An analysis is created when a workflow is run. It consists of some number of stages, each of which consists of either another analysis (if running a workflow) or a job (if running an app or applet).

  • Parent analysis: Each analysis is the parent analysis to each of the jobs that are created to run its stages.

• Job: A job is a unit of execution that is run on a worker in the cloud. A job is created when an app or applet is run, or when a job spawns another job.

  • Origin job: The job created when an app or applet is run by either a user or an analysis. An origin job always executes the "" entry point.

  • Master job: The job created when an app or applet is run by a user, job, or analysis. A master job always executes the "main" entry point. All origin jobs are also master jobs.
• Job-based object reference: A hash containing a job ID and an output field name. This hash is given in the input or output of a job. Once the specified job has transitioned to the "done" state, it is replaced with the specified job's output field.

This example demonstrates how to run TensorBoard inside a DNAnexus applet.

TensorBoard is a web application used to visualize and inspect what is going on inside TensorFlow training. To use TensorBoard, your training script in TensorFlow must include code that saves specific data to a log directory where TensorBoard can then find the data to display it.

This example uses an example script from the TensorBoard authors. For more guidance on how to use TensorBoard, check out the TensorFlow website ().

Creating the web application

Source Code

Step 1. Download BAM Files

Creating the web application

Inside the dxapp.json, you would add "httpsApp": {"ports":[443], "shared_access": "VIEW"} to tell the worker to expose this port.

R Shiny needs two scripts, server.R

Being notified of when a job may be stuck can help users to troubleshoot problems. On DNAnexus, users can set to limit the amount of time their jobs can run, or set a threshold on how long a job can take to run before the user is notified. The notification threshold can be specified in the executable at compile time via or .

When the threshold is reached for a , the system sends an email notification to both the user who launched the executable and the org admin.

When to Use Histograms

Histograms can be used to visualize numerical, date, and datetime data.

When to Use Scatter Plots

Scatter plots can be used to compare the distribution of values in a field containing numerical data, across different groups in a cohort. In a scatter plot, each such group is defined by its members sharing the same value in another field that also contains numerical data.

Primary field values are plotted on the x axis. Secondary field values are plotted on the y axis.

Overview

The CSV Loader ingests CSV files into a database. The input CSV files are loaded into a Parquet-format database and tables that can be queried using Spark SQL.

Overview

It may be necessary to preprocess, or harmonize, the data before you load them.

Harmonizing Data

Types of Cost and Spending Limits

A running execution can be terminated when it incurs charges that cause a cost or spending limit to be reached. When a spending limit is reached, this can also prevent new executions from being launched.

Execution Cost Limits

. This limit is set when a root execution is launched. Once this limit is reached, the DNAnexus Platform terminates running executions in the affected execution tree.

Errors

When an execution is terminated in this fashion, the Platform sets . This failure code is displayed on the UI, on the relevant project's Monitor page.

Billing Account Spending Limits

Billing account spending limits are managed by billing administrators, and can impact executions in projects billed to the account.

Billing account spending limits apply to cumulative charges incurred by projects billed to the account.

If cumulative charges reach this limit, the Platform terminates running jobs in projects billed to the account, and prevents new executions from being launched.

Errors

When a job is terminated in this fashion, the Platform sets as the failure reason. This failure reason is displayed on the UI, on the relevant project's Monitor page.

Project-Level Compute and Egress Spending Limits

, and can impact executions run within the project. Project admins can also set a separate monthly project-level egress spending limit, which can impact data egress from the project.

If the compute spending limit is reached, the Platform may terminate running jobs launched by project members, and prevent new executions from being launched. If the egress spending limit is reached, the Platform may prevent data egress from the project. The exact behavior depends on the policies of the org to which the project is billed.

For more information on these limits, see the , and the .

Compute Charges Incurred by Using Relational Database Clusters

Monthly project compute limits do not apply to compute charges incurred by using .

Compute Charges for Using Public IPv4 Addresses for Workers

Using public IPv4 addresses for workers incurs additional charges. When a job uses such a worker, IPv4 charges are included in the total cost figure shown for the job on the UI. These charges also count toward any .

For information on how to find the per-hour charge for using IPv4 addresses, in each cloud region in which org members can run executions, see the .

Getting Info on Cost and Spending Limits

Execution Costs and Cost Limits

The UI displays information on costs and cost limits for both individual executions and execution trees. Navigate to the project in which the execution or execution tree is being run, then click the Monitor tab. Click on the name of the execution or execution tree to open a page showing detailed information about it.

While an execution or execution tree is running, information is displayed on the charges it has incurred so far, and on additional charges it can incur, before an applicable cost limit is reached.

Spending Limits

Org spending limit information is available from the .

Project-Level Monthly Spending Limits

If project-level monthly spending limits have been set for a project, detailed information is available via the CLI, using the command .

While working in the Cohort Browser, you can visualize data using a variety of different types of charts.

To visualize data stored in particular field, follow these directions to browse through the fields in a dataset, select one, then create a chart based on the values in the field. When you select a field, the Cohort Browser suggests a chart type to use, to visualize the type of data it contains. You can also create multi-variable charts, displaying data from two fields, to help clarify the relationship between the data stored in each.

Single-Variable Charts

The following single-variable chart types are available in the Cohort Browser:

Multi-Variable Charts

The following multi-variable chart types are available in the Cohort Browser:

Interpreting Chart Data

Chart Totals and Missing Data

In all charts used in the Cohort Browser, a chart total count is displayed under the chart's title. This figure represents the number of records for which data is displayed in the chart. The label - "Participants" in the chart shown below - indicates the entity to which the data relates.

This figure is not always the same as the number of records in the cohort.

In a single-variable chart, if a field in a record is empty or contains a null value, that record is not included in the total, as its data can't be visualized. If any such records exist in the cohort, an "i" warning icon appears next to the chart total figure. Hover over the icon to show a tooltip with information about records that aren't included in the total.

The same holds for multi-variable charts. If any record contains a null value in either of the selected fields, or if either field is empty, that record isn't included in the chart total count, as its data can't be visualized.

When to Use Stacked Row Charts

Stacked row charts can be used to compare the distribution of values in a field containing categorical data, across different groups in a cohort. In a stacked row chart, each such group is defined by its patient sharing the same value in another field that also contains categorical data.

When creating a stacked row chart:

• Both the primary and secondary fields must contain categorical data

• Both the primary and secondary fields must contain no more than 20 distinct category values

Using Stacked Row Charts in the Cohort Browser

In the stacked row chart below, the primary field is VisitType, while DoctorType is the secondary field. In this chart, a cohort has been broken down into two groups, with the first sharing the value "Out-patient" in the VisitType field, while the second shares the value "In-patient."

The size of each bar, and the number to its right, indicate the total number of records in each group. In the chart below, for example, you can see that 3,179 records contain the value "Out-patient" in the VisitType field.

Each bar contains a color-coded section indicating how many of the group's records contain a specific value in the secondary field. Hovering over one of these sections reveals how many records, within a particular group, share a particular value in the secondary field. In the chart below, for example, you can see that 87 records in the first group share the value "specialist" in the DoctorType field.

Cohort Compare

Stacked row charts are not supported in Cohort Compare. Use a instead.

Preparing Data for Visualization in Stacked Row Charts

When , the following data types can be visualized in stacked row charts:

• String Categorical

• String Categorical Sparse

• Integer Categorical

Building a Kaplan-Meier Survival Curve Chart

To generate a survival chart, select one numerical field representing time, and one categorical field, which is transformed into the individual's status.

The categorical field should use one of the following 4 terms (case-insensitive) to indicate a status of "Living": living, alive, diseasefree, disease-free.

For multi-entity datasets, survival curve charts only support data fields from the main entity, or entities with 1:1 relation to the main entity.

Calculating Survival Percentage

To calculate survival percent at the current event the system evaluates the following formula:

• : Survival at the current event

• : Number of subjects living at the start of the period or event

• : Number of subjects that died

For each time period the following values are generated:

• Status: Each individual is considered Dead unless they qualify as Living.

• Number of Subjects Living at the Start ()

  • For the initial value this is the total number of records returned by the backend from survival data with Living or Dead Status.

This is the actual point drawn on the survival plot.

Learn More

View full source code on GitHub

This applet performs a basic samtools view -c {bam} command, referred to as "SAMtools count", on the DNAnexus Platform.

Download BAM Files

For bash scripts, inputs to a job execution become environment variables. The inputs from the dxapp.json file are formatted as shown below:

The object mappings_bam, a DNAnexus link containing the file ID of that file, is available as an environmental variable in the applet's execution. Use the command dx download to download the BAM file. By default, downloading a file preserves the filename of the object on the platform.

SAMtools Count

Use the bash helper variable mappings_bam_name for file inputs. For these inputs, the DNAnexus Platform creates a bash variable [VARIABLE]_name that holds the platform filename. Because the file was downloaded with default parameters, the worker filename matches the platform filename. The helper variable [VARIABLE]_prefix contains the filename minus any suffixes specified in the input field patterns (for example, the platform removes the trailing .bam to create [VARIABLE]_prefix).

Upload Result

Use command to upload data to the platform. This uploads the file into the job container, a temporary project that holds onto files associated with the job. When running the command dx upload with the flag --brief, the command returns only the file ID.

Job containers are an integral part of the execution process. To learn more see .

Associate With Output

The output of an applet must be declared before the applet is even built. Looking back to the dxapp.json file, you see the following:

The applet declares a file type output named counts_txt. In the applet script, specify which file should be associated with the output counts_txt. On job completion, this file is copied from the temporary job container to the project that launched the job.

When to Use Row Charts

Row charts can be used to visualize categorical data.

When creating a row chart:

• The data must be from a field that contains either categorical or categorical multi-select data

• This field must contain no more than 20 distinct category values

• The values cannot be organized in a hierarchy

Supported Field Types

See if you need to visualize hierarchical categorical data.

When to Use List Views for Categorical Data

Row charts can't be used to visualize data in categorical fields that have a hierarchical structure. For this type of data, use a .

Row charts aren't supported in Cohort Compare mode. In Cohort Compare mode, row charts are converted to .

Using Stacked Row Charts for Multivariate Visualizations

Row charts can't be used to visualize data from more than one field. To visualize categorical data from two fields, you can use a .

Using Row Charts in the Cohort Browser

In a row chart, each row shows a single category value, along with the number of records - the "count" - in which that value appears in the selected field. Also shown is the percentage of total cohort records in which it appears - its "freq." or "frequency."

Below is a sample row chart showing the distribution of values in a field Salt added to food. In the current cohort selection of 100,000 participants, 27,979 records contain the value "Sometimes", which represents 27.98% of the current cohort size.

Preparing Data for Visualization in Row Charts

When , the following data types can be visualized in row charts, if category values are specified as such in the coding file used at ingestion:

• String Categorical

• String Categorical Sparse

• String Categorical Multi-select

• Integer Categorical

Types of Time Limits

On the DNAnexus Platform, executions are subject to two independent time limits: job timeouts, and execution tree expirations.

Job Timeouts

Each job has a . This setting denotes the maximum amount of "wall clock time" that the job can spend in the "running" state, that is, running on the DNAnexus Platform.

If the job is still running when this limit is reached, the job is terminated.

The default job timeout setting is 30 days, though . A job may be given a .

How Job Timeouts Work

As noted above, job timeouts only apply to the time a job spends in the "running" state.

Job timeouts do not apply to any time a job spends waiting to begin running - as, for example, when a job is waiting for inputs to become available.

Job timeouts also do not apply to the time a job may spend between exiting the "running" state, and entering the "done" state - as, for example, when it is waiting for subjobs to finish.

To learn more about timeouts, see .

Errors

If a job fails to complete running before reaching its timeout limit, it is terminated, with .

Execution Tree Expiration

Each job is part of an . All jobs in an execution tree must complete running within 30 days of the launch of the tree's .

After this limit has been reached, all jobs within the execution tree lose the ability to access the Platform.

If an execution tree is restarted, its timeout setting is not reset. Jobs in the tree lose Platform access 30 days after the initial launch (the first try) of the tree's root execution.

Errors

If an execution tree reaches its time limit, jobs in the tree may not fail right away. If such a job is waiting for inputs or outputs, or if it is running without accessing the Platform, it may remain in that state. Only when the job tries to access the Platform does it fail. Depending on the access pattern, .

Monitoring Time Limits

To see information on time limits for execution and execution trees:

1. Navigate to the project in which the execution or execution tree is being run.

2. Click the Monitor tab.

3. Click the name of the execution or execution tree to open a page showing detailed information on it.

If a time limit is approaching, a warning message provides information on when the limit is reached.

If a job is waiting for subjobs to finish, it is shown as running, but job timeout information is not displayed. Execution tree information continues to be displayed.

Cohort Browser is a visualization tool for exploring and filtering structured datasets. It provides an intuitive interface for creating visualizations, defining patient cohorts, and analyzing complex data.

Cohort Browser supports multiple types of datasets:

• Clinical and phenotypic data - Patient demographics, clinical measurements, and outcomes

• Germline variants - Inherited genetic variations

Creating the web application

After configuring an app with Dash, start the server on port 443.

Inside the dxapp.json, you would add "httpsApp": {"ports":[443], "shared_access": "VIEW"}

Create interactive visualizations and manage dashboard layouts in the Cohort Browser.

Managing Dashboards

When to Use Grouped Box Plots

Grouped box plots can be used to compare the distribution of values in a field containing numerical data, across different groups in a cohort. In a grouped box plot, each such group is defined by its members sharing the same value in another field that contains categorical data.

When creating a grouped box plot:

The command-line client and the client bindings use a set of environment variables to communicate with the API server and to store state on the current default project and directory. These settings are set when you run dx login and can be changed through other dx commands. To display the active settings in human-readable format, use the dx env command:

To print the bash commands for setting the environment variables to match what dx is using, you can run the same command with the --bash flag.

Running a dx command from the command-line does not (and cannot) overwrite your shell's environment variables. The environment variables are stored in the ~/.dnanexus_config/environment file.

When to Use Box Plots

Box plots can be used to visualize numerical data.

Developers new to the DNAnexus Platform may find it helpful to learn by doing. This page contains a collection of tutorials and examples intended to showcase common tasks and methodologies when creating an app(let) on the DNAnexus Platform. After reading through the tutorials and examples you should be able to develop app(let)s that:

• Run efficiently: use cloud computing methodologies.

• Are straightforward to debug: let developers understand and resolve issues.

• Use the scale of the cloud: take advantage of the DNAnexus Platform's flexibility

• Are straightforward to use: reduce support and enable collaboration.

If it's your first time developing an app(let), read the series. This series introduces terms and concepts that tutorials and examples build on.

These tutorials are not meant to show realistic everyday examples, but rather provide a strong starting point for app(let) developers. These tutorials showcase varied implementations of the SAMtools view command on the DNAnexus Platform.

Bash App(let) Tutorials

Bash app(let)s use dx-toolkit, the platform SDK, and the command-line interface along with common Bash practices to create bioinformatic pipelines in the cloud.

Python App(let) Tutorials

Python app(let)s make of use dx-toolkit's along with common Python modules such as to create bioinformatic pipelines in the cloud.

Web App(let) Tutorials

To create a web applet, you need access to Titan or Apollo features. Web applets can be made as either Python or Bash applets. The only difference is that they launch a web server and expose port 443 (for HTTPS) to allow a user to interact with that web application through a web browser.

Concurrent Computing Tutorials

A bit of terminology before starting the discussion of parallel and distributed computing paradigms on the DNAnexus Platform.

Many definitions and approaches exist for tackling the concept of parallelization and distributing workloads in the cloud (Here's a on the subject). To make the documentation easier to understand when discussing concurrent computing paradigms, this guide refers to:

• Parallel: Using multiple threads or logical cores to concurrently process a workload.

• Distributed: Using multiple machines (in this case, cloud instances) that communicate to concurrently process a workload.

Keep these formal definitions in mind as you read through the tutorials and learn how to compute concurrently on the DNAnexus Platform.

Parallel

Distributed

View full source code on GitHub

To take full advantage of the scalability that cloud computing offers, your scripts have to implement the correct methodologies. This applet tutorial shows you how to:

1. Install SAMtools

2. Download BAM file

3. Count regions in parallel

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an package in the dxapp.json runSpec.execDepends.

For additional information, refer to the .

Download BAM file

The dxpy.download_all_inputs() function downloads all input files into the /home/dnanexus/in directory. A folder is created for each input and the files are downloaded to that directory. For convenience, the dxpy.download_all_inputs function returns a dictionary containing the following keys:

• <var>_path (string): full absolute path to where the file was downloaded.

• <var>_name (string): name of the file, including extension.

• <var>_prefix (string): name of the file minus the longest matching pattern found in the

The path, name, and prefix key-value pattern is repeated for all applet file class inputs specified in the dxapp.json. In this example, the dictionary has the following key-value pairs:

Count Regions in Parallel

Before performing the parallel SAMtools count, determine the workload for each thread. The number of workers is arbitrarily set to 10 and the workload per thread is set to 1 chromosome at a time. Python offers multiple ways to achieve multithreaded processing. For the sake of simplicity, use , a wrapper around Python's threading module.

Each worker creates a string to be called in a subprocess.Popen call. The multiprocessing.dummy.Pool.map(<func>, <iterable>) function is used to call the helper function run_cmd for each string in the iterable of view commands. Because multithreaded processing is performed using subprocess.Popen, the process does not alert to any failed processes. Closed workers are verified in the verify_pool_status helper function.

Important: In this example, you use subprocess.Popen to process and verify results in verify_pool_status. In general, it is considered good practice to use Python's built-in subprocess convenience functions. In this case, subprocess.check_call would achieve the same goal.

Gather Results

Each worker returns a read count of only one region in the BAM file. Sum and output the results as the job output. The dx-toolkit Python SDK function is used to upload and generate a DXFile corresponding to the result file. For Python, job outputs have to be a dictionary of key-value pairs, with the keys being job output names as defined in the dxapp.json and the values being the output values for corresponding output classes. For files, the output type is a . The function is used to generate the appropriate DXLink value.

View full source code on GitHub

To take full advantage of the scalability that cloud computing offers, your scripts must implement the correct methodologies. This applet tutorial shows you how to:

1. Install SAMtools

2. Download BAM file

3. Split workload

4. Count regions in parallel

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an package in the dxapp.json runSpec.execDepends field.

Download Inputs

This applet downloads all inputs at once using dxpy.download_all_inputs:

Split workload

Using the Python multiprocessing module, you can split the workload into multiple processes for parallel execution:

With this pattern, you can quickly orchestrate jobs on a worker. For a more detailed overview of the multiprocessing module, visit the .

Specific helpers are created in the applet script to manage the workload. One helper you may have seen before is run_cmd. This function manages the subprocess calls:

Before the workload can be split, you need to identify the regions present in the BAM input file. This initial parsing is handled in the parse_sam_header_for_region function:

Once the workload is split and processing has started, wait and review the status of each Pool worker. Then, merge and output the results.

The run_cmd function returns a tuple containing the stdout, stderr, and exit code of the subprocess call. These outputs from the workers are parsed to determine whether the run failed or passed.

View full source code on GitHub

What does this applet do?

This applet performs a basic SAMtools count of alignments present in an input BAM.

Prerequisites

The app must have network access to the hostname where the git repository is located. In this example, access.network is set to:

To learn more about access and network fields see .

How is the SAMtools dependency added?

SAMtools is cloned and built from the repository. The following is a closer look at the dxapp.json file's runSpec.execDepends property:

The execDepends value is a JSON array of dependencies to resolve before the applet source code is run. In this applet, the git fetch dependencies for htslib and SAMtools are specified. Dependencies resolve in the order listed. Specify htslib first, before the SAMtools build_commands, because newer versions of SAMtools depend on htslib. An overview of each property in the git dependency:

• package_manager - Details the type of dependency and how to resolve. .

• url - Must point to the server containing the repository. In this case, a GitHub URL.

• tag/branch - Git tag/branch to fetch.

The build_commands are executed from the destdir. Use cd when appropriate.

How is SAMtools called in the src script?

Because "destdir": "/home/dnanexus" is set in dxapp.json, the git repository is cloned to the same directory from which the script executes. The example directory's structure:

The SAMtools command in the app script is samtools/samtools.

Applet Script

You can build SAMtools in a directory that is on the $PATH or add the binary directory to $PATH. Keep this in mind for your app(let) development.

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json runSpec.execDepends.

Debugging

The command set -e -x -o pipefail assists you in debugging this applet:

• -e causes the shell to immediately exit if a command returns a non-zero exit code.

• -x prints commands as they are executed, which is useful for tracking the job's status or pinpointing the exact execution failure.

• -o pipefail makes the return code the first non-zero exit code. (Typically, the return code of pipes is the exit code of the last command, which can create difficult to debug problems.)

The *.bai file was an optional job input. You can check for a empty or unset var using the bash built-in test [[ - z ${var}} ]]. You can then download or create a *.bai index as needed.

Parallel Run

Bash's system allows for convenient management of multiple processes. In this example, bash commands are run in the background as the maximum job executions are controlled in the foreground. You can place processes in the background using the character & after a command.

Job Output

Once the input BAM has been sliced, counted, and summed, the output counts_txt is uploaded using the command . The following directory structure required for dx-upload-all-outputs is below:

In your applet, upload all outputs by creating the output directory and then using dx-upload-all-outputs to upload the output files.

View full source code on GitHub

How is Pysam provided?

Pysam is provided through a pip3 install using the pip3 package manager in the dxapp.json's runSpec.execDepends property:

The execDepends value is a JSON array of dependencies to resolve before the applet source code is run. In this applet, pip3 is specified as the package manager and pysam version 0.15.4 as the dependency to resolve.

Downloading Input

The fields mappings_sorted_bam and mappings_sorted_bai are passed to the main function as parameters for the job. These parameters are dictionary objects with key-value pair {"$dnanexus_link": "<file>-<xxxx>"}. File objects from the platform are handled through handles. If an index file is not supplied, then a *.bai index is created.

Working with Pysam

Pysam provides key methods that mimic SAMtools commands. In this applet example, the focus is only on canonical chromosomes. The Pysam object representation of a BAM file is pysam.AlignmentFile.

The helper function get_chr

Once a list of canonical chromosomes is established, you can iterate over them and perform the Pysam version of samtools view -c, pysam.AlignmentFile.count.

Uploading Outputs

The summarized counts are returned as the job output. The dx-toolkit Python SDK function uploads and generates a DXFile corresponding to the tabulated result file.

Python job outputs have to be a dictionary of key-value pairs, with the keys being job output names as defined in the dxapp.json file and the values being the output values for corresponding output classes. For files, the output type is a DXLink. The function generates the appropriate DXLink value.

View full source code on GitHub

How is the SAMtools dependency provided?

The SAMtools compiled binary is placed directory in the <applet dir>/resources directory. Any files found in the resources/ directory are uploaded so that they are present in the root directory of the worker. In this case:

When this applet is run on a worker, the resources/ folder is placed in the worker's root directory /:

/usr/bin is part of the $PATH variable, so the script can reference the samtools command directly, for example, samtools view -c ....

Parallel Run

Splice BAM

First, download the BAM file and slice it by canonical chromosome, writing the *bam file names to another file.

To split a BAM by regions, you need a *.bai index. You can either create an app(let) which takes the *.bai as an input or generate a *.bai in the applet. In this tutorial, you generate the *.bai in the applet, sorting the BAM if necessary.

Xargs SAMtools view

In the previous section, you recorded the name of each sliced BAM file into a record file. Next, perform a samtools view -c on each slice using the record file as input.

Upload results

The results file is uploaded using the standard bash process:

1. Upload a file to the job execution's container.

2. Provide the DNAnexus link as a job's output using the script dx-jobutil-add-output <output name>

Overview

VCF Loader ingests Variant Call Format (VCF) files into a database. The input VCF files are loaded into a Parquet-format database that can be queried using Spark SQL.

The input VCF for every run can be a single VCF file or many VCF files, but the merged input must represent a single logical VCF file. In the many files case, the logical VCF may be partitioned by chromosome, by genomic region, and/or by sample. In any case, every input VCF file must be a syntactically correct, sorted VCF file.

VCF Preprocessing

Although VCF data can be loaded into Apollo databases after the variant call step, the dataset may not be normalized for downstream analyses across large cohorts. In that case, preprocessing and harmonizing the data before loading is recommended. To learn more, see .

How to Run VCF Loader

Input:

• vcf_manifest: (file) a text file containing a list of file ID's of the VCF files to load (one per line). The referenced files' names must be distinct and end in .vcf.gz. If more than one file is specified, then the complete VCF file to load is considered to be partitioned and every specified partition must be a valid VCF file. After the partition-merge step in preprocessing, the complete VCF file must be valid.

Required Parameters:

• database_name: (string) name of the database into which to load the VCF files.

• create_mode: (string) strict mode creates database and tables from scratch and optimistic mode creates databases and tables if they do not already exist.

Other Options:

• snpeff: (boolean) default true -- whether to include the SnpEff annotation step in preprocessing with INFO/ANN tags. If SnpEff annotations are desired in the database, then either pre-annotate the raw VCF separately, or include this SnpEff annotation step -- it is not necessary to do both.

• snpeff_human_genome: (string) default GRCh38.92 -- id of the SnpEff human genome to use in the SnpEff annotation step in preprocessing.

Basic Run

To take full advantage of the scalability that cloud computing offers, your scripts have to implement the correct methodologies. This applet tutorial shows you how to:

1. Install SAMtools

2. Download BAM file

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json file's runSpec.execDepends.

For additional information, see execDepends.

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an package in the dxapp.json runSpec.execDepends.

How is the SAMtools dependency provided?

The SAMtools compiled binary is placed directory in the <applet dir>/resources directory. Any files found in the resources/ directory are uploaded so that they are present in the root directory of the worker. In this case:

When this applet is run on a worker, the

How is the SAMtools dependency provided?

The SAMtools dependency is resolved by declaring an Apt-Get package in the dxapp.json file's runSpec.execDepends.

For additional information, see execDepends

Extracting Data From a Dataset With Spark

The dx

Explore and analyze datasets with germline data by opening them in the Cohort Browser and switching to the Germline Variants tab. You can create cohorts based on germline variants, visualize variant patterns, and examine detailed variant information.

Filtering by Germline Variants

You can to include only samples with specific germline variants.

To apply a germline filter to your cohort:

Explore and analyze datasets with gene expression assays by opening them in the Cohort Browser and switching to the Gene Expression tab. You can create cohorts based on expression levels, visualize expression patterns, and examine detailed gene information.

DNAnexus allows organizations to optionally reuse outputs of jobs that share the same executable and input IDs, even if these outputs are across projects or entire organizations. This feature has two primary use cases.