DNAnexus Documentation
APIDownloadsIndex of dx CommandsLegal
  • Overview
  • Getting Started
    • DNAnexus Essentials
    • Key Concepts
      • Projects
      • Organizations
      • Apps and Workflows
    • User Interface Quickstart
    • Command Line Quickstart
    • Developer Quickstart
    • Developer Tutorials
      • Bash
        • Bash Helpers
        • Distributed by Chr (sh)
        • Distributed by Region (sh)
        • SAMtools count
        • TensorBoard Example Web App
        • Git Dependency
        • Mkfifo and dx cat
        • Parallel by Region (sh)
        • Parallel xargs by Chr
        • Precompiled Binary
        • R Shiny Example Web App
      • Python
        • Dash Example Web App
        • Distributed by Region (py)
        • Parallel by Chr (py)
        • Parallel by Region (py)
        • Pysam
      • Web App(let) Tutorials
        • Dash Example Web App
        • TensorBoard Example Web App
      • Concurrent Computing Tutorials
        • Distributed
          • Distributed by Region (sh)
          • Distributed by Chr (sh)
          • Distributed by Region (py)
        • Parallel
          • Parallel by Chr (py)
          • Parallel by Region (py)
          • Parallel by Region (sh)
          • Parallel xargs by Chr
  • User
    • Login and Logout
    • Projects
      • Project Navigation
      • Path Resolution
    • Running Apps and Workflows
      • Running Apps and Applets
      • Running Workflows
      • Running Nextflow Pipelines
      • Running Batch Jobs
      • Monitoring Executions
      • Job Notifications
      • Job Lifecycle
      • Executions and Time Limits
      • Executions and Cost and Spending Limits
      • Smart Reuse (Job Reuse)
      • Apps and Workflows Glossary
      • Tools List
    • Cohort Browser
      • Chart Types
        • Row Chart
        • Histogram
        • Box Plot
        • List View
        • Grouped Box Plot
        • Stacked Row Chart
        • Scatter Plot
        • Kaplan-Meier Survival Curve
      • Locus Details Page
    • Using DXJupyterLab
      • DXJupyterLab Quickstart
      • Running DXJupyterLab
        • FreeSurfer in DXJupyterLab
      • Spark Cluster-Enabled DXJupyterLab
        • Exploring and Querying Datasets
      • Stata in DXJupyterLab
      • Running Older Versions of DXJupyterLab
      • DXJupyterLab Reference
    • Using Spark
      • Apollo Apps
      • Connect to Thrift
      • Example Applications
        • CSV Loader
        • SQL Runner
        • VCF Loader
      • VCF Preprocessing
    • Environment Variables
    • Objects
      • Describing Data Objects
      • Searching Data Objects
      • Visualizing Data
      • Filtering Objects and Jobs
      • Archiving Files
      • Relational Database Clusters
      • Symlinks
      • Uploading and Downloading Files
        • Small File Sets
          • dx upload
          • dx download
        • Batch
          • Upload Agent
          • Download Agent
    • Platform IDs
    • Organization Member Guide
    • Index of dx commands
  • Developer
    • Developing Portable Pipelines
      • dxCompiler
    • Cloud Workstation
    • Apps
      • Introduction to Building Apps
      • App Build Process
      • Advanced Applet Tutorial
      • Bash Apps
      • Python Apps
      • Spark Apps
        • Table Exporter
        • DX Spark Submit Utility
      • HTTPS Apps
        • Isolated Browsing for HTTPS Apps
      • Transitioning from Applets to Apps
      • Third Party and Community Apps
        • Community App Guidelines
        • Third Party App Style Guide
        • Third Party App Publishing Checklist
      • App Metadata
      • App Permissions
      • App Execution Environment
        • Connecting to Jobs
      • Dependency Management
        • Asset Build Process
        • Docker Images
        • Python package installation in Ubuntu 24.04 AEE
      • Job Identity Tokens for Access to Clouds and Third-Party Services
      • Enabling Web Application Users to Log In with DNAnexus Credentials
      • Types of Errors
    • Workflows
      • Importing Workflows
      • Introduction to Building Workflows
      • Building and Running Workflows
      • Workflow Build Process
      • Versioning and Publishing Global Workflows
      • Workflow Metadata
    • Ingesting Data
      • Molecular Expression Assay Loader
        • Common Errors
        • Example Usage
        • Example Input
      • Data Model Loader
        • Data Ingestion Key Steps
        • Ingestion Data Types
        • Data Files Used by the Data Model Loader
        • Troubleshooting
      • Dataset Extender
        • Using Dataset Extender
    • Dataset Management
      • Rebase Cohorts and Dashboards
      • Assay Dataset Merger
      • Clinical Dataset Merger
    • Apollo Datasets
      • Dataset Versions
      • Cohorts
    • Creating Custom Viewers
    • Client Libraries
      • Support for Python 3
    • Walkthroughs
      • Creating a Mixed Phenotypic Assay Dataset
      • Guide for Ingesting a Simple Four Table Dataset
    • DNAnexus API
      • Entity IDs
      • Protocols
      • Authentication
      • Regions
      • Nonces
      • Users
      • Organizations
      • OIDC Clients
      • Data Containers
        • Folders and Deletion
        • Cloning
        • Project API Methods
        • Project Permissions and Sharing
      • Data Object Lifecycle
        • Types
        • Object Details
        • Visibility
      • Data Object Metadata
        • Name
        • Properties
        • Tags
      • Data Object Classes
        • Records
        • Files
        • Databases
        • Drives
        • DBClusters
      • Running Analyses
        • I/O and Run Specifications
        • Instance Types
        • Job Input and Output
        • Applets and Entry Points
        • Apps
        • Workflows and Analyses
        • Global Workflows
        • Containers for Execution
      • Search
      • System Methods
      • Directory of API Methods
      • DNAnexus Service Limits
  • Administrator
    • Billing
    • Org Management
    • Single Sign-On
    • Audit Trail
    • Integrating with External Services
    • Portal Setup
    • GxP
      • Controlled Tool Access (allowed executables)
  • Science Corner
    • Scientific Guides
      • Somatic Small Variant and CNV Discovery Workflow Walkthrough
      • SAIGE GWAS Walkthrough
      • LocusZoom DNAnexus App
      • Human Reference Genomes
    • Using Hail to Analyze Genomic Data
    • Open-Source Tools by DNAnexus Scientists
    • Using IGV Locally with DNAnexus
  • Downloads
  • FAQs
    • EOL Documentation
      • Python 3 Support and Python 2 End of Life (EOL)
    • Automating Analysis Workflow
    • Backups of Customer Data
    • Developing Apps and Applets
    • Importing Data
    • Platform Uptime
    • Legal and Compliance
    • Sharing and Collaboration
    • Product Version Numbering
  • Release Notes
  • Technical Support
  • Legal
Powered by GitBook

Copyright 2025 DNAnexus

On this page
  • Example Execution Tree
  • Job States
  • Successful Jobs
  • Unsuccessful Jobs
  • Terminated Jobs
  • Failed Jobs
  • Restartable Jobs
  • Additional States
  • Analysis States
  • Billing

Was this helpful?

Export as PDF
  1. User
  2. Running Apps and Workflows

Job Lifecycle

Learn about the states through which a job or analysis may go, during its lifecycle.

Last updated 11 months ago

Was this helpful?

Example Execution Tree

In the following example, we have a workflow that has two stages, one of which is an applet, and the other of which is an app.

If the workflow is run, it will generate an analysis with an attached workspace for storing intermediate output from its stages. Jobs are also created to run the two stages. These jobs in turn can spawn more jobs, either to run another function in the same executable or to run an executable. The blue labels indicate which jobs or analyses can be described using a particular term (as defined above).

Job States

Successful Jobs

Jobs may also pass through the following transitional states as part of more complicated execution patterns:

  • waiting_on_input (between idle and runnable): a job enters and stays in this state if at least one of the following is true:

    • it has an unresolved job-based object reference in its input

    • it has a data object input that cannot be cloned yet because it is not in the closed state or a linked hidden object is not in the closed state

    • it was created to wait on a list of jobs or data objects that must enter the done or closed states, respectively (see the dependsOn field of any API call that creates a job); linked hidden objects are implicitly included in this list

  • waiting_on_output (between running and done): a job enters and stays in this state if at least one of the following is true:

    • it has a descendant job that has not been moved to the done state

    • it has an unresolved job-based object reference in its output

    • it is an origin or master job which has a data object (or linked hidden data object) output in the closing state

Unsuccessful Jobs

There are two terminal job states other than the done state, terminated and failed, and a job can enter either of these states from any other state except another terminal state.

Terminated Jobs

The terminated state is entered when a user has requested that the job (or another job that shares the same origin job) be terminated. For all terminated jobs, the failureReason in their describe hash will be set to "Terminated", and the failureMessage will indicate the user responsible for terminating the job. Only the user who launched the job or administrators of the job's project context can terminate the job.

Failed Jobs

On the DNAnexus Platform, jobs are limited to a runtime of 30 days. Jobs running longer than 30 days will fail with JobTimeoutExceeded error.

Restartable Jobs

Jobs belonging to root executions launched after July 12, 2023 00:13 UTC have a try integer attribute representing different tries for restarted jobs. The first try of a job has try set to 0. The second job try (if the job was restarted) has its try attribute set to 1, and so on.

  • restartable: a job in a restartable state indicates that the job is ready to be restarted.

  • restarted: a job try in a restarted state indicates that the job try was restarted.

Additional States

For unsuccessful jobs, there are a couple more states that jobs may enter between the running state and its eventual terminal state of terminated or failed; unsuccessful jobs starting in all other non-terminal states will be transitioned directly to the appropriate terminal state.

  • terminating: the transitional state when the worker in the cloud has begun terminating the job and tearing down the execution environment. Once the worker in the cloud has reported that it has terminated the job or otherwise becomes unresponsive, then the job will transition to its terminal state.

Analysis States

All analyses start in the state in_progress, and, like jobs, will end up in one of the terminal states done, failed, or terminated. The following diagram shows the state transition for all successful analyses.

If an analysis is unsuccessful, it may transition through one or more intermediate states before it reaches its terminal state:

  • partially_failed: this state indicates that one or more stages in the analysis have not finished successfully, and there is at least one stage which has not transitioned to a terminal state. In this state, some stages may have already finished successfully (and entered the done state), and the remaining stages will also be allowed to finish successfully if they can.

  • terminating: an analysis may enter this state either via an API call where a user has terminated the analysis, or there is some failure condition under which the analysis is terminating any remaining stages. This may happen if the executionPolicy for the analysis (or a stage of an analysis) had the onNonRestartableFailure value set to "failAllStages".

Billing

In general, compute and data storage costs due to jobs that end up failing because of user error (e.g. InputError, OutputError) and terminated jobs are still charged to the project in which the jobs were run. For internal errors of the DNAnexus platform, such costs will not be billed.

The costs for each stage in an analysis is determined independently. If the first stage finishes successfully while a second stage fails for a system error, the first stage will still be billed, and the second will not.

Note that the subjob or child job of stage 1's origin job shares the same temporary workspace as its parent job. Any calls to run a new applet or app (using the API methods or will launch a master job that has its own separate workspace, and (by default) no visibility into its parent job's workspace.

Every successful job goes through at least the following four states: 1. idle: initial state of every new job, regardless of what API call was made to create it. 2. runnable: the job's are ready, and it is not waiting for any other job to finish or data object to finish closing. 3. running: the job has been assigned to and is being run on a worker in the cloud. 4. done: the job has completed, and it is not waiting for any descendent job to finish or data object to finish closing. This is a terminal state, so no job will become a different state after transitioning to done.

Jobs can fail for a variety of reasons, and once a job fails, this triggers failure for all other jobs that share the same origin job. If an unrelated job (i.e. is not in the same job tree) has a job-based object reference or otherwise depends on a failed job, then it will also fail. For more information about errors that jobs can encounter, see the .

Jobs can automatically restart upon certain types of failures, such as UnresponsiveWorker, ExecutionError, AppInternalError, JobTimeoutExceeded, which can be specified in the executionPolicy of an app(let) or workflow. If a job fails for a restartable reason, its failure propagates to its nearest master job and restarts that job (if that executable has flag set to the default value of master) or restarts the job itself (if the if the executable has flag set to the default value of all ). A job can be restarted the number of times that is given in the executionPolicy, after which the entire job tree fails.

Some ( e.g. , , , , , , ) accept optional job try input and may include job's try attribute in their output. All API methods interpret job ID inputs without a try argument as referring to the most recent try corresponding to that job ID.

debug_hold: a job has been run with debugging options and has failed for an applicable reason, and is being held for debugging by the user. For more information about triggering this state, see the .

inputs
Error Information page
Connecting to Jobs page
/app-xxxx/run
API methods
restartableEntryPoints
restartableEntryPoints
/system/findExecutions
/system/findJobs
/system/findAnalyses
/applet-xxxx/run
/job-xxxx/describe
/job-xxxx/addTags
/job-xxxx/removeTags
/job-xxxx/setProperties