Monitoring Executions
Learn how to get information on current and past executions, via both the UI and the CLI.
Last updated
Was this helpful?
Learn how to get information on current and past executions, via both the UI and the CLI.
Last updated
Was this helpful?
To get basic information on :
Click on Projects in the main Platform menu.
On the Projects list page, find and click on the name of the project within which the execution was launched.
Click on the Monitor tab to open the Monitor screen.
On the Monitor screen, you'll see a list of executions launched within the project. By default, they're listed in reverse chronological order, with the most recently launched execution at the top.
Find the row displaying information on the execution.
Note that for an analysis (the execution of a workflow), you can click on the "+" icon to the left of the analysis name to expand the row to view information on its stages. If an execution has further descendants, you can click on the “+” icon next to its name to expand the row further.
To see additional information on an execution, .
Note that there are shortcuts you can use to view information that is found on the details page directly on the list page, or relaunch an execution:
To view the :
Click the Info icon, above the right edge of the executions list, if it’s not already selected, and then select the execution by clicking on the row, or
Hover over the row and click on the “More Actions” button that looks like three vertical dots at the end of the row to select View Info in the fly out menu.
To view the for a job, do either of the following:
Select the execution by clicking on the row. A View Log button will appear in the header. Click the View Log button, or
Hover over the row and click on the “More Actions” button that looks like three vertical dots at the end of the row to select View Log in the fly out menu.
To , do either of the following:
Select the execution by clicking on the row. A Launch as New Job button will appear in the header. Click the Launch as New Job button, or
Hover over the row and click on the “More Actions” button that looks like three vertical dots at the end of the row, then select Launch as New Job in the flyout menu.
To , do either of the following:
Select the execution by clicking on the row. A Launch as New Analysis button will appear in the header. Click the Launch as New Analysis button, or
Hover over the row and click on the “More Actions” button that looks like three vertical dots at the end of the row to select Launch as New Analysis in the flyout menu.
In the list on the Monitor screen, you'll see the following information for each of the executions that is running or has been run within the project in question:
State - This is the execution's state. State values include:
"Waiting" - When initially launched, the execution's state will be "Waiting" until the Platform has allocated the resources required to run it, and, in some cases, until other executions on which it depends have finished.
"Running" - Once a job has started to run, its state will change from "Waiting" to “Running.”
"In Progress" - Once an analysis has been launched, its state will change to "In Progress."
"Done" - If the execution completes with no errors, its state will change to "Done."
"Terminating" - If the worker has begun terminating an execution, but not yet finished doing so, the execution's state will be displayed as "Terminating."
"Terminated" - If the execution is terminated prior to completion, its state will change to "Terminated."
"Debug Hold" - If an execution has been run with debugging options, and has failed for an applicable reason, and is being held for debugging, its state will show as "Debug Hold."
Executable - The executable or executables run in the course of the execution. Note that if the execution is an analysis, each stage will be shown in a separate row, including the name of the executable run during the stage in question. Note as well that if there is an informational page giving details about the executable and how to configure and use it, the executable's name will be clickable, and clicking the name will display that page.
Launched By - The name of the user who launched the execution.
Launched On - The time at which the execution was launched. Note that for many executions, this will be earlier than the time displayed in the Started Running column, because many executions spend time waiting for resources to become available, before they start running.
Started Running - The time at which the execution started running, if it has done so. Note that this is not always the same as its launch time, if it has to spend time waiting for resources to become available, before it can start running.
Duration - For jobs, this figure represents the time elapsed since the job entered the running state. For analyses, it represents the time elapsed since the analysis was created.
Cost - A value is displayed in this column when the user has access to billing info for the execution. The figure shown represents either, for a running execution, an estimate of the charges it has incurred so far, or, for a completed execution, the total costs it incurred.
Output Folder - For each execution, the value displayed represents a path relative to the project's root folder. Clicking the value will open the folder in which the execution's outputs have been or will be stored.
Additional basic information can be displayed for each execution. To do this:
Click on the "table" icon at the right edge of the table header row.
Select one or more of the entries in the list, to display an additional column or columns.
Available additional columns include:
Stopped Running - The time at which the execution stopped running.
To remove columns from the list, click on the "table" icon at the right edge of the table header row, then de-select one or more of the entries in the list, to hide the column or columns in question.
A filter menu above the executions list allows you to run a search that refines the list to display only executions meeting specific criteria.
By default, pills are displayed that allow you to set search criteria that will filter executions by one or more of the following attributes:
Name - Execution name
State - Execution state
Executable - A specific executable
Launched By - The user who launched an execution or executions
Launch Time - The time range within which executions were launched
Click the List icon, above the right edge of the executions list, to display pills that allow filtering by additional execution attributes.
To save a particular filter, click the Bookmark icon, above the right edge of the executions list, assign your filter a name, then click Save.
To apply a saved filter to the executions list, click the Bookmark icon, then select the filter from the list.
As a launcher of a given execution, and a contributor to the project within which the execution is running, you can terminate the execution from the list on the Monitor screen, when it's in a non-terminal state. You can also terminate executions launched by other project members if you have project admin status.
To terminate an execution:
Find the execution in the list:
Select the execution by clicking on the row. A red Terminate button will appear at the end of the header. Click the Terminate button; or
Hover over the row and click on the “More Actions” button that looks like three vertical dots at the end of the row to select Terminate in the fly out menu.
A modal window will open, asking you to confirm that you want to terminate the execution. Click Terminate to confirm.
The execution's state will show as "Terminating" as it is being terminated. Then its state will change to "Terminated."
To get additional information on an execution, click on its name in the list on the Monitor screen. A new page will open.
On the details page for an execution, you'll see a range of information, including:
High-level details - In the Execution Tree section, at the top of the screen, you'll see high-level information, including:
For a standalone execution - such as a job without children - you'll see a single entry that includes details on the state of the execution, when it started and stopped running, and how long it spent in the running state.
For an execution with descendants - such as an analysis with multiple stages - you'll see a list, with each row containing details on the execution run at each stage of the analysis. If the execution has descendants, you can click on the “+” icon next to its name to expand the row to view information on its descendants. To see a page displaying detailed information on a stage, click on its name in the list. To navigate back to the workflow's details page, click on its name in the "breadcrumb" navigation menu in the top right corner of the screen.
Execution state - In the Execution Tree section, each execution row includes a color bar that represents the execution's current state. For descendants within the same execution tree, the time visualizations are staggered, indicating their different start and stop times in relation to each other. The colors include:
Blue - A blue bar indicates that the execution is in the "Running" or "In Progress" state.
Green - A green bar indicates that the execution is in the "Done" state.
Red - A red bar indicates that the execution is in the "Failed" or "Partially Failed" state.
"Grey" indicates that the execution is in the "Terminated" state.
Execution start and stop times - Times are displayed in the header bar at the top of the Execution Tree section. These times run, from left to right, from the time at which the job started running, or when the analysis was created, to either the current time, or the time at which the execution entered a terminal state ("Done," "Failed," or "Terminated").
Inputs - In this section, you'll see a list of the inputs to the execution. If a direct link to the input file is available, the input's name will be hyperlinked to the file; clicking the link will open the project location containing the file. If the input was provided by another execution in a workflow, the execution's name will be hyperlinked; clicking the link will open the details page for the execution in question.
Outputs - In this section, you'll see a list of the execution's outputs. If a direct link to the output file is available, the output's name will be hyperlinked to the file; clicking the link will open the folder containing the file.
Log files - An execution's log file is useful in understanding details about, for example, the resources used by an execution, the costs it incurred, and the source of any delays it encountered. To access log files, and, as needed, download them in .txt format:
To access the log file for a job, click either the View Log button in the top right corner of the screen, or the View Log link in the Execution Tree section.
To access the log file for each stage in an analysis, click the View Log link next to the row displaying information on the stage in question, in the Execution Tree section.
Reused results - If an execution reuses results from another execution, this information will be shown in a blue pane, above the Execution Tree section. To see details on the execution that generated these results, click on its name.
If an execution failed, a Cause of Failure pane will display, above the Execution Tree section. The cause of failure is a system-generated error message. For assistance in diagnosing the failure and any related issues:
Click the button labeled Send Failure Report to DNAnexus Support.
A form will open in a modal window, with both the Subject and Message fields pre-populated with information that DNAnexus Support will use in diagnosing and resolving the issue.
By clicking the button in the Grant Access section, DNAnexus Support reps will be given "View" access to the project in which the issue occurred. This will enable Support reps to diagnose and resolve the issue more quickly.
Click Send Report to send the report.
To re-launch a job from the execution details screen:
Click the Launch as New Job button in the upper right corner of the screen.
A new browser tab will open, displaying the Run App / Applet form.
Configure the run, then click Start Analysis.
To re-launch an analysis from the execution details screen:
Click the Launch as New Analysis button in the upper right corner of the screen.
A new browser tab will open, displaying the Run Analysis form.
Configure the run, then click Start Analysis.
If you want to save a copy of a workflow along with its input configurations under a new name, from the execution details screen:
Click the Save as New Workflow button in the upper right corner of the screen.
In the Save as New Workflow modal window, give the workflow a name, and select the project in which you'd like to save it.
Click Save.
If you want to view the execution details for the initial tries for a restarted job:
Click on the "Tries" link below the job name in the summary banner, or the "Tries" link next to the job name in the execution tree.
A modal window will open.
Click the name of the try for which you'd like to view execution details.
You can use dx watch to view the log of a running job or any past jobs, which may have finished successfully, failed, or been terminated.
You can use dx find executions to return the ten most recent executions in your current project. You can specify the number of executions you wish to view by running dx find executions -n <specified number>. The output from dx find executions will be similar to the information shown in the "Monitor" tab on the DNAnexus web UI.
Below is an example of dx find executions; in this case, only two executions have been run in the current project. There is an individual job, DeepVariant Germline Variant Caller, and a workflow consisting of two stages, Variant Calling Workflow. A stage is represented by either another analysis (if running a workflow) or a job (if running an app(let)).
The job running the DeepVariant Germline Variant Caller executable is running and has been running for 10 minutes and 28 seconds. The analysis running the Variant Calling Workflow consists of 2 stages, Freebayes Variant Caller, which is waiting on input, and BWA-MEM FASTQ Read Mapper, which has been running for 10 minutes and 18 seconds.
By default, the dx find executions operation will search for jobs or analyses created when a user runs an app or applet. If a job is part of an analysis, the results will be returned in a tree representation linking all of the jobs in an analysis together.
By default, dx find executions will return up to ten of the most recent executions in your current project in order of execution creation time.
However, a user can also filter the returned executions by job type. Using the flag --origin-jobs in conjunction with the dx find executions command returns only original jobs, whereas the flag --all-jobs will also include subjobs.
We can choose to monitor only analyses by running the command dx find analyses. Analyses are executions of workflows and consist of one or more app(let)s being run. When using dx find analyses, the command will return only the top-level analyses, not any of the jobs contained therein.
Below is an example of dx find analyses:
Below is an example of dx find jobs:
Searches for executions can be restricted to specific parameters.
To extract stdout only from this job, we can run the command dx watch job-xxxx --get-stdout
To extract stderr only from this job, we can run the command dx watch job-xxxx --get-stderr\
To extract only stdout and stderr from this job, we can run the command dx watch job-xxxx --get-streams
Below is an example of viewing stdout lines of a job log:
To view the entire job tree, including both main jobs and subjobs, use the command dx watch job-xxxx --tree.
To view the entire job tree -- both main jobs and subjobs -- use the command dx watch job-xxxx -n 8. If the job already ran, the output is displayed as well.
In the example below, the app Sample Prints doesn’t have any output.
By default, dx find will restrict your search to only your current project context. To search across all the projects to which you have access, use the --all-projects flag.
By default, dx find will only return up to ten of the most recently launched executions matching your search query. To change the number of executions returned, you can use the -n option.
Users can also use the --created-before and --created-after options to search based on when the execution began.
Users can also restrict the search to a specific state, e.g. "done", "failed", "terminated".
The --delim flag will tab-delimit the output. This allows the output to be passed into other shell commands.
You can use the --brief flag to return only the object IDs for the objects returned by your search query. The ‑‑origin‑jobs flag will omit the subjob information.
Below is an example usage of the --brief flag:
Below is an example of using the flags --origin-jobs and --brief. In the example below, we describe the last job run in the current default project.
Name - The default name for an execution is the name of the app, applet, or workflow being run. When configuring an execution, you can give it a custom name, either , or . Note that the execution's name is used in Platform email alerts related to the execution. Note as well that clicking on a name in the executions list opens the .
"Failed" - If the execution fails to complete with errors, its state will change to "Failed." See for help in understanding why an execution failed.
"Partially Failed" - if one or more stages in the workflow have not finished successfully, and there is at least one stage that has not transitioned to a terminal state (either "Done," "Failed," or "Terminated").
Tags - Tags are strings associated with objects on the platform. They are a .
Priority - The priority assigned to the execution - either "low," "normal," or "high" - when it was configured, either or . This setting determines the scheduling priority of the execution, vis-a-vis other executions that are waiting to be launched.
Worker URL - If the execution is running an executable - such as - to which you can connect directly via a web URL, that URL will be shown here. Clicking the URL will open a connection to the executable, in a new browser tab.
Custom properties columns - If a have been assigned to any of the listed executions, a column can be added to the table, for each such property, showing the values assigned to each execution, for that property.
ID - An execution's or
Note that by default, filters are set to display only that meet the criteria defined in the filter. If you want the display to include all executions, including those run during individual stages of workflows, click the button, above the left edge of the executions list, showing the default value "Root Executions Only." Then click "All Executions."
Basic info - The Info pane, on the right side of the screen, displays a on the execution, along with additional detail such as the execution's unique ID, and .
As described in , jobs can be configured to restart automatically upon certain types of failures.
Note that you can only for the most recent try, not for any previous tries.
If you'd like to view the job's log stream while it runs, you can use . The log stream includes a log of stdout, stderr, and additional information the worker outputs as it executes the job.
If for some reason you need to terminate a job before it completes, use the command .
If you'd like to view any jobs that have finished running, you can use the command. The log stream includes a log of stdout, stderr, and additional information the worker outputs as it executed the job.
Jobs are runs of an individual app(let) and compose analyses. We can monitor jobs by running the command , which will return a flat list of jobs. If a job is in an analysis, all jobs within the analysis are also returned.
Jobs can be configured to restart automatically upon certain types of failures as described in the section. To view initial tries of the restarted jobs along with execution subtrees rooted in those initial tries, use dx find executions --include-restarted. To examine job logs for initial tries, use dx watch job-xxxx --try X. An example of these commands is shown below.
A user can search for only executions of a specific app(let) or workflow based on its .
See the for more on using dx find jobs.
A license is required to use this feature. for more information.
Job logs can be automatically forwarded to a customer's Splunk instance for analysis. See for more information on enabling and using this feature.