DNAnexus Documentation
APIDownloadsIndex of dx CommandsLegal
  • Overview
  • Getting Started
    • DNAnexus Essentials
    • Key Concepts
      • Projects
      • Organizations
      • Apps and Workflows
    • User Interface Quickstart
    • Command Line Quickstart
    • Developer Quickstart
    • Developer Tutorials
      • Bash
        • Bash Helpers
        • Distributed by Chr (sh)
        • Distributed by Region (sh)
        • SAMtools count
        • TensorBoard Example Web App
        • Git Dependency
        • Mkfifo and dx cat
        • Parallel by Region (sh)
        • Parallel xargs by Chr
        • Precompiled Binary
        • R Shiny Example Web App
      • Python
        • Dash Example Web App
        • Distributed by Region (py)
        • Parallel by Chr (py)
        • Parallel by Region (py)
        • Pysam
      • Web App(let) Tutorials
        • Dash Example Web App
        • TensorBoard Example Web App
      • Concurrent Computing Tutorials
        • Distributed
          • Distributed by Region (sh)
          • Distributed by Chr (sh)
          • Distributed by Region (py)
        • Parallel
          • Parallel by Chr (py)
          • Parallel by Region (py)
          • Parallel by Region (sh)
          • Parallel xargs by Chr
  • User
    • Login and Logout
    • Projects
      • Project Navigation
      • Path Resolution
    • Running Apps and Workflows
      • Running Apps and Applets
      • Running Workflows
      • Running Nextflow Pipelines
      • Running Batch Jobs
      • Monitoring Executions
      • Job Notifications
      • Job Lifecycle
      • Executions and Time Limits
      • Executions and Cost and Spending Limits
      • Smart Reuse (Job Reuse)
      • Apps and Workflows Glossary
      • Tools List
    • Cohort Browser
      • Chart Types
        • Row Chart
        • Histogram
        • Box Plot
        • List View
        • Grouped Box Plot
        • Stacked Row Chart
        • Scatter Plot
        • Kaplan-Meier Survival Curve
      • Locus Details Page
    • Using DXJupyterLab
      • DXJupyterLab Quickstart
      • Running DXJupyterLab
        • FreeSurfer in DXJupyterLab
      • Spark Cluster-Enabled DXJupyterLab
        • Exploring and Querying Datasets
      • Stata in DXJupyterLab
      • Running Older Versions of DXJupyterLab
      • DXJupyterLab Reference
    • Using Spark
      • Apollo Apps
      • Connect to Thrift
      • Example Applications
        • CSV Loader
        • SQL Runner
        • VCF Loader
      • VCF Preprocessing
    • Environment Variables
    • Objects
      • Describing Data Objects
      • Searching Data Objects
      • Visualizing Data
      • Filtering Objects and Jobs
      • Archiving Files
      • Relational Database Clusters
      • Symlinks
      • Uploading and Downloading Files
        • Small File Sets
          • dx upload
          • dx download
        • Batch
          • Upload Agent
          • Download Agent
    • Platform IDs
    • Organization Member Guide
    • Index of dx commands
  • Developer
    • Developing Portable Pipelines
      • dxCompiler
    • Cloud Workstation
    • Apps
      • Introduction to Building Apps
      • App Build Process
      • Advanced Applet Tutorial
      • Bash Apps
      • Python Apps
      • Spark Apps
        • Table Exporter
        • DX Spark Submit Utility
      • HTTPS Apps
        • Isolated Browsing for HTTPS Apps
      • Transitioning from Applets to Apps
      • Third Party and Community Apps
        • Community App Guidelines
        • Third Party App Style Guide
        • Third Party App Publishing Checklist
      • App Metadata
      • App Permissions
      • App Execution Environment
        • Connecting to Jobs
      • Dependency Management
        • Asset Build Process
        • Docker Images
        • Python package installation in Ubuntu 24.04 AEE
      • Job Identity Tokens for Access to Clouds and Third-Party Services
      • Enabling Web Application Users to Log In with DNAnexus Credentials
      • Types of Errors
    • Workflows
      • Importing Workflows
      • Introduction to Building Workflows
      • Building and Running Workflows
      • Workflow Build Process
      • Versioning and Publishing Global Workflows
      • Workflow Metadata
    • Ingesting Data
      • Molecular Expression Assay Loader
        • Common Errors
        • Example Usage
        • Example Input
      • Data Model Loader
        • Data Ingestion Key Steps
        • Ingestion Data Types
        • Data Files Used by the Data Model Loader
        • Troubleshooting
      • Dataset Extender
        • Using Dataset Extender
    • Dataset Management
      • Rebase Cohorts and Dashboards
      • Assay Dataset Merger
      • Clinical Dataset Merger
    • Apollo Datasets
      • Dataset Versions
      • Cohorts
    • Creating Custom Viewers
    • Client Libraries
      • Support for Python 3
    • Walkthroughs
      • Creating a Mixed Phenotypic Assay Dataset
      • Guide for Ingesting a Simple Four Table Dataset
    • DNAnexus API
      • Entity IDs
      • Protocols
      • Authentication
      • Regions
      • Nonces
      • Users
      • Organizations
      • OIDC Clients
      • Data Containers
        • Folders and Deletion
        • Cloning
        • Project API Methods
        • Project Permissions and Sharing
      • Data Object Lifecycle
        • Types
        • Object Details
        • Visibility
      • Data Object Metadata
        • Name
        • Properties
        • Tags
      • Data Object Classes
        • Records
        • Files
        • Databases
        • Drives
        • DBClusters
      • Running Analyses
        • I/O and Run Specifications
        • Instance Types
        • Job Input and Output
        • Applets and Entry Points
        • Apps
        • Workflows and Analyses
        • Global Workflows
        • Containers for Execution
      • Search
      • System Methods
      • Directory of API Methods
      • DNAnexus Service Limits
  • Administrator
    • Billing
    • Org Management
    • Single Sign-On
    • Audit Trail
    • Integrating with External Services
    • Portal Setup
    • GxP
      • Controlled Tool Access (allowed executables)
  • Science Corner
    • Scientific Guides
      • Somatic Small Variant and CNV Discovery Workflow Walkthrough
      • SAIGE GWAS Walkthrough
      • LocusZoom DNAnexus App
      • Human Reference Genomes
    • Using Hail to Analyze Genomic Data
    • Open-Source Tools by DNAnexus Scientists
    • Using IGV Locally with DNAnexus
  • Downloads
  • FAQs
    • EOL Documentation
      • Python 3 Support and Python 2 End of Life (EOL)
    • Automating Analysis Workflow
    • Backups of Customer Data
    • Developing Apps and Applets
    • Importing Data
    • Platform Uptime
    • Legal and Compliance
    • Sharing and Collaboration
    • Product Version Numbering
  • Release Notes
  • Technical Support
  • Legal
Powered by GitBook

Copyright 2025 DNAnexus

On this page
  • Connecting to Jobs via SSH
  • Setting Up Your Environment for SSH Access
  • Connecting to Jobs via SSH
  • Connecting to Spark Cluster Workers via SSH
  • Setting Up SSH on Windows
  • Using PuTTY
  • API Details
  • Debug Hold

Was this helpful?

Export as PDF
  1. Developer
  2. Apps
  3. App Execution Environment

Connecting to Jobs

Last updated 1 year ago

Was this helpful?

Note: As of dx v0.319.2 or later, dx ssh is able to access a job started without the --allow-ssh flag, by adding the client IP address to the list of IP addresses that are allowed to connect to the job. When using dx run --allow-ssh / --ssh, ssh access is now more secure and the default is to allow only connections from your client IP address. To use these features, upgrade to dx v0.319.2 or later with ​​pip3 install --upgrade dxpyor by following DNAnexus .

Connecting to Jobs via SSH

Jobs running on the DNAnexus Platform can be optionally configured to allow SSH connections to the DNAnexus worker executing the job. This can be used to monitor the job underway, to troubleshoot a failed job, or to employ the worker as a workstation in the cloud.

By default, jobs are firewalled from the Internet, and only have network access to the DNAnexus API. Outbound access can be configured using . Inbound access is restricted to SSH connectivity only, and must be enabled explicitly by the user at run time.

Logging and Reproducibility Guarantees

A quick note about reproducibility guarantees: the DNAnexus platform provides features in support of high-level reproducibility of analyses. In general, apps given the same inputs produce the same outputs when run again. Features such as network access (other than to the DNAnexus API) are incompatible with this reproducibility support. Please be aware that when you connect to a job, reproducibility can no longer be ensured.

Setting Up Your Environment for SSH Access

One-time setup of the user's account is required to allow use of SSH connections. Open a command shell and use dx ssh_config to perform this setup.

  • If you haven't set up a public SSH key on the machine you're using before, use option 0 ("Generate a new SSH key pair using ssh-keygen").

  • If you have previously set up a public SSH key on the machine you're using, select one of the SSH key pairs listed, or option 0 if you wish to generate a new key pair.

  • If you have previously set up a public SSH key on another computer, you'll have to copy the keys into your current environment. The keys are found (by default) in ~/.dnanexus_config as ssh_id and ssh_id.pub.

If you already have a key and you generate a new one, the old one will be overwritten. Note that you won't be able to SSH into jobs from any computer that has an older key.

Connecting to Jobs via SSH

Once you have setup your ssh keypair, you can connect to individual jobs you launch on DNAnexus.

SSH connections to jobs can be established by running dx ssh job-xxxx.

Alternatively, you can launch a job and immediately ssh into that job by running: dx run executable --ssh.

Connecting to Spark Cluster Workers via SSH

To connect to the worker nodes in a cluster job, first connect via SSH to the driver node which will make sure your client IP is in the allowedSSH list of the cluster job. You can then connect to the first cluster worker node by running the following from your local machine (not the driver node). ssh -i ~/.dnanexus_config/ssh_id dnanexus@$(dx describe --json --verbose job-xxxx|jq -rc .clusterSlaves[0].host) where clusterSlaves[0] refers to the first worker node.

Setting Up SSH on Windows

Once OpenSSH is installed, start DX CLI ( Start menu | DNAnexus CLI (folder)| DNAnexus CLI (shortcut) ), then run dx login or dx login --token <token>(if you have token created from the profile settings via the web inteface previoussly) to log into the platform. After logged in, run dx ssh_config and choose 0 to generate a new ssh key pair. This will create the private key "ssh_id" file and the public key "ssh_id.pub" file under the ".dnanexus_config" folder within your home.

You should then be able to follow the instructions above.

Using PuTTY

  1. Use PuTTYgen to import your private key. You can do this by:

    1. Clicking on "Load" in PuTTYgen, changing the filetype from "PuTTY Private Key Files" to "All Files", and selecting the ssh_id file found in the .dnanexus_config folder inside of your Windows User folder.

    2. Then click on "Save private key" and save as "ssh_id.ppk".

  2. Make sure allowedSSH field of your job includes your client IP address, as seen by the ssh daemon on the worker.

    1. If you are not using ssh proxy, you can find your client IP by running dx api system whoami '{"fields":{"clientIp":true}}' |jq .clientIp

    2. If you are using SSH proxy, the client IP address as seen by the SSH daemon on the worker will be external IP address of the SSH proxy.

    3. If the output of the command: dx describe job-xxxx --json |jq .allowSSH does not contain your client IP address, add it to the job's allowSSH field with dx api job-xxxx update '{"allowSSH":["<clientIP>"]}'

  3. Get the URL of the job you would like to ssh into by running dx describe job-xxxx and making note of the url found in the host field.

  4. In PuTTY:

    a. Place the URL found above in the "Host Name" field.

    b. In Connection->Data, place dnanexus in the "Auto-login username" field.

    c. In Connection->SSH->Auth, click "Browse" to select your "ssh_id.ppk" file you generated in Step 1.

    d. In Window->Translation->Remote character set: change the setting from UTF-8 to ISO-8859-1: 1998 (Latin-1, West Europe).

    e. In Terminal setting, uncheck the first setting "Auto wrap mode initially on" checkbox.

    f. In Terminal->Keyboard-> The Function keys and keypad, check Xterm R6 radio box.

    g. Save the session by giving a name of your choice in Session->Saved Sessions field, then click on the "Save" button on the right.

  5. Click the Open button at the bottom of the screen to login to your job.

Setting Up SSH Access via Proxy

If you would like to use a proxy during SSH connection, run dx ssh --ssh-proxy <proxy_address>:<proxy_port> or dx run --ssh --ssh-proxy <proxy_address>:<proxy_port>. For dx run with SSH access, you must specify both --ssh and --ssh-proxy at runtime. If you do not specify the proxy with dx run --ssh, you will have to exit out of the current SSH session and initiate a new session using dx ssh --ssh-proxy <proxy_address>:<proxy_port> to access the job via proxy.

API Details

The following details describe the API functionality used by the high-level dx commands described above.

  • The job was launched by that user.

  • The job's allowSSH field is set and includes a hostmask that matches the IP address from which the connection is being made.

  • The SSH client uses a private key that matches the public key stored in their user account.

  • The job was started by running an applet, an open-source app, or a non-open-source app of which the user is listed as a developer.

Note that SSH connections to non-open source apps are not supported, if the user is not listed as a developer of the app in question.

If a job is terminated while you are logged in via SSH, your connection will be terminated as well.

Debug Hold

Jobs can be optionally configured to hold the execution environment for debugging when certain types of errors happen (debug hold). When jobs are held for debugging, the user can log in to their execution environment using dx ssh (as long as it has been configured via dx ssh_config).

When entering debug hold, the job transitions into the debug_hold state, and any extra information about the failure reason is set in its debug.failureReason and debug.failureMessage fields. Jobs in this state are held for up to 2 days, and terminated by the system afterwards. Jobs in debug hold can also be terminated by the user either using the API (e.g. via dx terminate), or by terminating the process tree in the job's execution environment. In all cases, the job can only transition into the terminated or failed state.

When a job's debug.debugOn field contains one of the eligible failure reasons (AppError, AppInternalError, or ExecutionError, typically specified via dx run --debug-on All), the job can enter debug hold via the following two conditions:

  1. The job can quit with a non-zero exit status. In this case the job's original process tree is replaced with a sentinel process.

  2. The job can create the file named /.dx-hold. This will induce the execution environment to transition into debug hold without altering the job's process tree, allowing any running processes to remain intact.

Another good use of ssh features is in debugging failing jobs. The platform allows you to specify conditions under which a failed job should be held for debugging using dx run --debug-on. If the job encounters an error, the worker will not be terminated as usual, but will remain open for 72 hours, giving the user time to ssh into the worker and debug the underlying issue. See the section below.

Finally, a great use of ssh features on the platform is to launch a cloud workstation. For more information about configuring a job to employ a worker as a workstation in the cloud, please see our .

Running dx ssh job-xxxx will connect you to the driver node of a job.

If you are using Windows, we highly recommend installing , an open source connectivity tool for remote login with the SSH protocol.

If you would like to use as your ssh terminal, you will first need to generate your public/private key pair as described above. Once that is done, you can do the following:

SSH connection authentication is supported using SSH keys only. To configure a user's account to allow SSH connections to jobs, the sshPublicKey field is passed to . This field is returned in only when called by that user; other users cannot see the user's SSH public key.

To configure a job to allow SSH connections, the allowSSH field is passed to the , , , or method. A user can connect to a job via SSH only when all of the following conditions are satisfied:

Once connected, the terminal uses a byobu () or a tmux () terminal window manager, allowing the terminal to persist even if you get disconnected. Further information on how to use the terminal is presented in a banner when you log in.

docs
network access permissions
tutorial on cloud workstations
Spark cluster
OpenSSH
PuTTY
external link
external link
Debug Hold
/applet-xxxx/run
/job-xxxx/update
/user-xxxx/update
/user-xxxx/describe
/app-xxxx/run
/workflow-xxxx/run