> For the complete documentation index, see [llms.txt](https://documentation.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.dnanexus.com/user/running-apps-and-workflows/job-lifecycle.md).

# Job Lifecycle

Learn about the states through which a job or analysis may go, during its lifecycle.

## Example Execution Tree

The following example shows a workflow that has two stages, one of which is an applet, and the other of which is an app.

```mermaid
flowchart TD
    subgraph WF["Workflow A"]
        direction TB
        S0["Stage 0: Applet X"]
        S1["Stage 1: App Y"]
        S0 ~~~ S1
    end

    %% Styling
    classDef workflowGroup fill:#F6F7FA,stroke:#121249,stroke-width:2px,color:#121249
    classDef stageNode fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249

    class WF workflowGroup
    class S0,S1 stageNode
```

When the workflow runs, it generates an analysis with an attached workspace for storing intermediate output from its stages. Jobs are created to run the two stages. These jobs can spawn additional jobs to run other functions in the same executable or to run separate executables. The blue labels indicate which jobs or analyses each term refers to.

```mermaid
flowchart TD
    subgraph analysisWS["Analysis workspace"]
        analysis["<b>analysis</b><br/>executable: workflow A"]
        rootExec["root execution"]
        parentAnalysis["parent analysis"]
        analysis ~~~ rootExec ~~~ parentAnalysis
    end
    subgraph zWS["Job workspace"]
        zjob["<b>job</b><br/>executable: applet Z<br/>function: main"]
        zm["master job"]
        zjob ~~~ zm
    end
    subgraph stage1WS["Job workspace"]
        stage1["<b>Stage 1: job</b><br/>executable: app Y<br/>function: main"]
        o1["origin job"]
        m1["master job"]
        p1["parent job"]
        subjob["<b>job</b><br/>executable: app Y<br/>function: process"]
        sub["subjob"]
        stage1 ~~~ o1 ~~~ m1 ~~~ p1
        p1 --> subjob
        subjob ~~~ sub
    end
    subgraph stage0WS["Job workspace"]
        stage0["<b>Stage 0: job</b><br/>executable: applet X<br/>function: main"]
        o0["origin job"]
        m0["master job"]
        p0["parent job"]
        stage0 ~~~ o0 ~~~ m0 ~~~ p0
    end
    analysis --> stage0
    analysis --> stage1
    stage1 --> zjob

    %% Styling
    classDef workspaceGroup fill:#F6F7FA,stroke:#121249,stroke-width:2px,color:#121249
    classDef jobNode fill:#FFFFFF,stroke:#121249,stroke-width:2px,color:#121249
    classDef analysisNode fill:#FFDC04,stroke:#121249,stroke-width:2px,color:#121249
    classDef tagNode fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249

    class analysisWS,stage0WS,stage1WS,zWS workspaceGroup
    class analysis analysisNode
    class stage0,stage1,subjob,zjob jobNode
    class rootExec,parentAnalysis,o0,m0,p0,o1,m1,p1,sub,zm tagNode
```

The *subjob* or *child job* of stage 1's origin job shares the same temporary workspace as its parent job. API calls to run a new applet or app using [`/applet-xxxx/run`](https://documentation.dnanexus.com/developer/api/running-analyses/applets-and-entry-points.md#api-method-applet-xxxx-run) or [`/app-xxxx/run`](https://documentation.dnanexus.com/developer/api/running-analyses/apps.md) launch a *master job* that has its own separate workspace, and (by default) no visibility into its parent job's workspace.

## Job States

### Successful Jobs

Every successful job goes through at least the following four states:

1. **idle**: initial state of every new job, regardless of what API call was made to create it.
2. **runnable**: the job's inputs are ready, and it is not waiting for any other job to finish or for a data object to finish closing.
3. **running**: the job has been assigned to and is being run on a worker in the cloud.
4. **done**: the job has completed, and it is not waiting for any descendent job to finish or for a data object to finish closing. This is a terminal state, so no job becomes a different state after transitioning to **done**.

```mermaid
flowchart LR
    idle --> runnable --> running --> done

    %% Styling
    classDef activeState fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249
    classDef doneState fill:#61BA00,stroke:#121249,stroke-width:2px,color:#121249

    class idle,runnable,running activeState
    class done doneState
```

Jobs may also pass through the following transitional states as part of more complicated execution patterns:

* **`waiting_on_input`** (between **idle** and **runnable**): a job enters and stays in this state if at least one of the following is true:
  * it has an unresolved job-based object reference in its input
  * it has a data object input that cannot be cloned yet because it is not in the **closed** state or a linked hidden object is not in the **closed** state
  * it was created to wait on a list of jobs or data objects that must enter the **done** or **closed** states, respectively (see the `dependsOn` field of any API call that creates a job). Linked hidden objects are implicitly included in this list
* **`waiting_on_output`** (between **running** and **done**): a job enters and stays in this state if at least one of the following is true:
  * it has a descendant job that has not been moved to the **done** state
  * it has an unresolved job-based object reference in its output
  * it is an origin or master job which has a data object (or linked hidden data object) output in the **closing** state

```mermaid
flowchart LR
    idle -.-> runnable --> running -.-> done
    idle -.-> waiting_on_input -.-> runnable
    running -.-> waiting_on_output -.-> done

    %% Styling
    classDef activeState fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249
    classDef transitionState fill:#FFDC04,stroke:#121249,stroke-width:2px,color:#121249
    classDef doneState fill:#61BA00,stroke:#121249,stroke-width:2px,color:#121249

    class idle,runnable,running activeState
    class waiting_on_input,waiting_on_output transitionState
    class done doneState
```

### Unsuccessful Jobs

Two terminal job states exist other than the **done** state: **terminated** and **failed**. A job can enter either of these states from any other state *except* another terminal state.

### Terminated Jobs

The **terminated** state occurs when a user requests termination of the job (or another job sharing the same origin job). For all terminated jobs, the `failureReason` in their describe hash contains "Terminated", and the `failureMessage` indicates the user responsible for termination. Only the user who launched the job or administrators of the job's project context can terminate the job.

### Failed Jobs

Jobs can fail for a variety of reasons, and once a job fails, this triggers failure for all other jobs that share the same origin job. If an unrelated job not in the same job tree has a job-based object reference or otherwise depends on a failed job, then it also fails. For more information about errors that jobs can encounter, see the [Error Information page](https://documentation.dnanexus.com/developer/apps/error-information.md).

{% hint style="warning" %}
On the DNAnexus Platform, jobs are limited to a runtime of 30 days. Jobs running longer than 30 days fail with a `JobTimeoutExceeded` error.
{% endhint %}

### Restartable Jobs

Jobs can automatically restart when they encounter specific types of failures. You configure which failure types trigger restarts in the `executionPolicy` of an app, applet, or workflow. Common restartable failure types include:

* `UnresponsiveWorker`
* `ExecutionError`
* `AppInternalError`
* `AppInsufficientResourceError`
* `JobTimeoutExceeded`
* `SpotInstanceInterruption`

#### How job restarts work

When a job fails for a restartable reason, the system determines where to restart based on the [`restartableEntryPoints`](https://documentation.dnanexus.com/developer/apps/io-and-run-specifications.md#run-specification) configuration:

* **`master` setting (default):** The failure propagates to the nearest master job, which then restarts
* **`all` setting:** The job restarts itself directly

The system restarts a job up to the maximum number of times specified in the `executionPolicy`. Once this limit is reached, the entire job tree fails.

During the restart process, jobs transition through specific states:

* **restartable:** The job is ready to be restarted
* **restarted:** The job attempt was restarted (a new attempt begins)

#### Job try tracking

For jobs in root executions launched after July 12, 2023 00:13 UTC, the platform tracks restart attempts using a `try` integer attribute:

* First attempt: `try` = 0
* Second attempt (first restart): `try` = 1
* Third attempt (second restart): `try` = 2

Multiple [API methods](https://documentation.dnanexus.com/developer/api/api-directory.md) support job try operations and include try information in their responses:

* `/job-xxxx/describe`
* `/job-xxxx/addTags`
* `/job-xxxx/removeTags`
* `/job-xxxx/setProperties`
* `/system/findExecutions`
* `/system/findJobs`
* `/system/findAnalyses`

When you provide a job ID without specifying a `try` argument, these methods automatically refer to the most recent attempt for that job.

### Additional States

For unsuccessful jobs, additional states exist between the **running** state and the terminal state of **terminated** or **failed**. Unsuccessful jobs starting in other non-terminal states transition directly to the appropriate terminal state.

* **terminating**: the transitional state when the cloud worker begins terminating the job and tearing down the execution environment. The job moves to its terminal state after the worker reports successful termination or becomes unresponsive.
* **`debug_hold`**: a job has been run with debugging options and has failed for an applicable reason, and is being held for debugging by the user. For more information about triggering this state, see the [Connecting to Jobs page](https://documentation.dnanexus.com/developer/apps/connecting-to-jobs.md).

```mermaid
flowchart LR
    running --> terminating & debug_hold
    terminating --> failed & terminated
    debug_hold --> failed & terminated

    %% Styling
    classDef activeState fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249
    classDef transitionState fill:#FFDC04,stroke:#121249,stroke-width:2px,color:#121249
    classDef failState fill:#FF2C94,stroke:#121249,stroke-width:2px,color:#121249

    class running activeState
    class terminating,debug_hold transitionState
    class failed,terminated failState
```

## Analysis States

All analyses start in the state **`in_progress`**, and, like jobs, reach one of the terminal states **done**, **failed**, or **terminated**. The following diagram shows the state transitions for successful analyses.

```mermaid
flowchart LR
    in_progress --> done

    %% Styling
    classDef activeState fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249
    classDef doneState fill:#61BA00,stroke:#121249,stroke-width:2px,color:#121249

    class in_progress activeState
    class done doneState
```

If an analysis is unsuccessful, it may transition through one or more intermediate states before it reaches its terminal state:

* `partially_failed`: this state indicates that one or more stages in the analysis have not finished successfully, and there is at least one stage which has not transitioned to a terminal state. In this state, some stages may have already finished successfully (and entered the **done** state), and the remaining stages are also allowed to finish successfully if they can.
* `terminating`: an analysis may enter this state either via an API call where a user has terminated the analysis or because there is some failure condition under which the analysis is terminating any remaining stages. This may happen if the `executionPolicy` for the analysis (or a stage of an analysis) had the `onNonRestartableFailure` value set to "failAllStages".

```mermaid
flowchart LR
    in_progress --- j1(( )) --- j2(( )) --- j3(( ))
    j1 --> partially_failed --> j2
    j2 --> terminating --> j3
    j3 --> failed & terminated

    %% Styling
    classDef activeState fill:#18ADF1,stroke:#121249,stroke-width:2px,color:#121249
    classDef transitionState fill:#FFDC04,stroke:#121249,stroke-width:2px,color:#121249
    classDef failState fill:#FF2C94,stroke:#121249,stroke-width:2px,color:#121249
    classDef junctionNode fill:#121249,stroke:#121249,stroke-width:2px,color:#FFFFFF

    class in_progress activeState
    class partially_failed,terminating transitionState
    class failed,terminated failState
    class j1,j2,j3 junctionNode
```

## Billing

Compute and data storage costs for jobs that fail due to user error are charged to the project running those jobs. This includes errors such as `InputError` and `OutputError`. The same applies to terminated jobs. For DNAnexus Platform internal errors, these costs are not billed.

The costs for each stage in an analysis are determined independently. If the first stage finishes successfully while a second stage fails for a system error, the first stage is still billed, and the second is not.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://documentation.dnanexus.com/user/running-apps-and-workflows/job-lifecycle.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
