DNAnexus Platform SDK

The DNAnexus Platform SDK (a.k.a. dx-toolkit) includes the command-line client, tools for building and debugging apps, and other utilities for working with DNA data on the platform. Our SDK is released under the Apache v2.0 license.

Note: If you plan to do most of your interaction with the platform via the website, and will use the command-line only to upload files, you probably do not need to download the SDK. The Upload Agent (discussed below) is our specialized tool for uploading to the platform and may be all that you need from this page.

You can download the SDK built for your platform:

Install

1. Unpack the SDK:

   $ tar -xzf dx-toolkit*.tar.gz

   Note: Python 2.7 is required. Command-line tools in the SDK will not work with Python 2.6 or Python 3.

2. Initialize your environment as follows:

   $ source dx-toolkit/environment

   Congratulations! You are now ready to use the Platform SDK.

What's next:

• The Command Line Quickstart will get you started on accessing the platform from the command-line.

• The Developer section contains comprehensive information on how to use the SDK to interact with and write your own applications for the DNAnexus Platform.

Proxy and TLS Settings

To direct dx-toolkit to a proxy, please set the HTTP_PROXY environment variable to something like export HTTP_PROXY=hostname:port. HTTPS_PROXY is also supported.

By default, dx-toolkit uses certificates installed on the system to create secure connections. If your system requires an additional TLS certificate and the dx-toolkit doesn't appear to be using a certificate installed on your system, there are two options in order of preference. First, set the DX_CA_CERT environment variable to the path of the PEM-encoded TLS certificate file required by your parent organization. You can also connect insecurely by avoiding certificate verification all together by setting DX_CA_CERT=NOVERIFY.

For Windows users, the way environments are customized can vary enough to where the settings above may still cause issues. Please contact support@dnanexus.com if you are having trouble connecting through a proxy.

Building from source

Download either of the following source packages:

See the Readme.md file in the archive for build-time dependencies and further installation instructions (or view on Github).

Alternative downloads

pypi

The Python components of the SDK (dxpy bindings and most of the command-line tools) are distributed through PyPI. Use pip to download and install the dxpy package:

pip install dxpy

To enable tab completion after installing with pip, run the following command or add it to your .bashrc

eval "$(register-python-argcomplete dx|sed 's/-o default//')"

Note: OS X 10.9 users with Python < 2.7.6 from python.org or pythonbrew may encounter Python issue 18458 when using the Python interpreter interactively. Upgrade to Python 2.7.6 or greater to resolve the issue.

Ubuntu APT repository

If you are running Ubuntu 16.04 or 14.04, see Ubuntu Repository for instructions on how to obtain the DNAnexus SDK via APT.

Upgrading the DNAnexus SDK

The dx upgrade command is a useful tool to ensure that the users of the DNAnexus platform have the latest command-line tools and can customize their toolkit to suit the requirements of their project. This command will upgrade your DNAnexus SDK to the latest version recommended. The dx upgrade command can also revert your dx-toolkit to a previous version.

Calling dx upgrade without any arguments will upgrade your DNAnexus SDK to the most recent version. Alternatively, you can provide dx upgrade with a version of the DNAnexus SDK that you would prefer to have installed on your local system.

Upgrading to the latest version:

$ dx upgrade
Upgrading to v0.228.0
Downloading dx-toolkit-v0.286.1-osx.tar.gz (using curl -O -L)...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9874k  100 9874k    0     0  3279k      0  0:00:03  0:00:03 --:--:-- 3279k
Unpacking dx-toolkit-v0.286.1-osx.tar.gz...
Backing up version v0.227.1...
Moving version v0.228.0 into place...
upgrade.sh: Updated to version v0.228.0. Previous version saved in /Users/user-amy/dnanexus/dx-toolkit/build/v0.227.1.
Please close this terminal, open a new terminal, and source the environment file in the dx-toolkit folder

Upgrading to a previous version:

$ dx upgrade v0.225.0

Upload Agent

The Upload Agent is a command-line tool for uploading files to the DNAnexus Platform. It is a fast and convenient alternative to using the web interface for uploading files.

Features

• Fast Parallel Uploads: Data is uploaded using multiple parallel connections for faster upload speeds.

• Automatic Compression: Text files (such as FASTA, FASTQ, qseq, and others) can be automatically compressed before upload, for better efficiency.

• Automated Uploads: No user interaction is required; the client can be used to automate uploads of entire sequencing runs.

• Resumable Uploads: Previously interrupted uploads can be automatically resumed.

• Progress Reporting: You can monitor individual file upload progress and also average/instantaneous transfer rates.

Downloads

The Upload Agent is available pre-built for the three main flavors of operating systems:

Install

Unpack and run the downloaded file. You should then see a help message.

• 64-bit Linux:

  $ tar -xzf dnanexus-upload-agent-*-linux.tar.gz
  $ cd dnanexus-upload-agent-*-linux
  $ ./ua

• Mac OS X

  $ unzip dnanexus-upload-agent-*-osx.zip
  $ cd dnanexus-upload-agent-*-osx
  $ ./ua

• Windows

  • Unzip the file using your favorite utility

  • Run "ua.exe"

  Note for Windows 7/2008 users: If you receive an error message saying: 'schannel: failed to setup extended errors', please install this hotfix from Microsoft.

This program requires that you have an API key. Please create an API key before proceeding.

Download Agent

The Download Agent is a command-line tool for downloading large quantities of files from DNAnexus to a local environment. It is designed to resiliently manage downloads over potentially long periods of time.

Features and Usage

For a detailed description of Download Agent features and usage, please review the README file on its Github location.

Downloads

A list of Download Agent releases is provided on the Releases section of its Github location. The Docker repository additionally provides access access experimental builds corresponding to commits the master branch.

dxWDL

dxWDL is a compiler from the Workflow Description Language (WDL) to the platform. It takes a workflow written in WDL, and generates a DNAnexus native workflow from it. It can be downloaded from the releases section of the github repository. The executable is a jar file that can be run with java 8.

Troubleshooting

This section discusses common issues we have encountered when installing dx-toolkit on a variety of platforms.

Exception thrown when logging in

On October 15th, 2017 DNAnexus disabled support for TLS 1.0 and 1.1, and this is a common cause of login issues. Addressing these issues is typically system-specific. To see if your Python release supports TLS 1.2, you can run:

python2.7 -c "import ssl ; print(ssl.OPENSSL_VERSION)"

If that command prints an OpenSSL version that is less than 1.0.1, then you will need to take additional steps to obtain a Python release with TLS 1.2 support.

Linux

For Linux, the correct solution will depend on what distribution you are using. This article by Stripe provides a nice summary of how to install the required packages.

macOS / OS X

Recommended:

If you have the option of installing macOS High Sierra (10.13) on the machine, we recommend you perform this upgrade since Apple has added TLS 1.2 support to the SSL library used by the Python release that ships with the OS.

Alternative:

If you do not have the option of upgrading macOS, you can use homebrew to obtain a TLS 1.2-capable Python. Please go to https://brew.sh and follow their instructions to install homebrew.

Next, install Python:

brew install python

Inspect your path by running this command:

echo $PATH

to see if the dir /usr/local/opt/python/libexec/bin is present.

If it's not, you may need to add this line to your ~/.bash_profile file to insert homebrew's python bin directory at the head of your PATH:

export PATH="/usr/local/opt/python/libexec/bin:$PATH"

You can double-check the value that should be used in that export command by running brew info python and examining the export command that is printed in the "Caveats" section.