Workflows and Analyses
Last updated
Was this helpful?
Last updated
Was this helpful?
Workflows are objects which list a series of executables (apps or applets) and configuration parameters specifying how to run them. For example, a DNA sequencing workflow may consist of a series of 3 apps: mapping, variant calling, and variant annotation. The outputs of one executable can be configured to be inputs to the next. Each executable listed in a workflow, together with it configuration and I/O parameters, is called a stage. At the moment, workflows are not allowed to be used as executables for a stage.
An analysis is the execution of a workflow, similar to how a job is the execution of an app. Both jobs and analyses can also be referred to as the runs of their respective executables (workflows, apps, or applets).
To create a new workflow, use the API method. The workflow can be edited using various API methods that support a variety of edit actions. The workflow can then be run using the API method. This API method runs all the stages in the workflow and creates an analysis object, which contains metadata about all the jobs (and perhaps other analyses) that were created.
At any point after an analysis has been created, the workflow from which it was run can be recovered by calling with the field initializeFrom
set to the ID of the analysis.
For information on building or managing Nextflow workflows, see .
A workflow object has an edit version number which can be retrieved using the API call . It must be provided every time an API call is made to edit a workflow and must match the current value in order to succeed. The new edit version number is returned upon a successful edit.
You can specify what stages should be run when creating the workflow using ; you can also add additional stages after creating the workflow with . When adding a stage, you must specify the executable that will be run. You must also specify a unique stage ID. However, when adding stages to an already existing workflow with , the ID is optional, and if you do not supply it, a unique ID is generated on your behalf (see the section Stage ID and Name for more information).
This ID will be unique for the stage in the workflow; you will need to provide it when making further changes to the stage or cross-linking outputs and inputs of the stages.
Besides the executable that it runs, each stage can also have the following metadata:
Name
that are exported for the workflow
Stage ID and Name
Stage IDs must match the regular expression ^[a-zA-Z_][0-9a-zA-Z_-]{0,255}$
(only letters, numbers, underscores, and dashes, at least one char, does not start with a number or dash; maximum length is 256 characters).
The stage name is a non-unique label used for display purposes. It allows you to provide a descriptive identifier for the stage that will be shown in the UI in the workflow view. If not provided, the executable's name is displayed instead.
Stage Output Folders
The following table shows some examples for where a stage's output will go for different values of the stage's folder
field, under the condition that the workflow's output folder is "/foo":
Stage's folder
Value
Stage Output Folder
null (no value)
"/foo"
"bar/baz"
"/foo/bar/baz"
"/quux"
"/quux"
Workflow Input
One consequence of defining a workflow with an explicit input is that once the workflow is created, all the input values will need to be provided by the user to workflow inputs and not to stages. By linking stage inputs with workflow inputs during workflow build time, all the values provided to a workflow-level input (here reference_genome
) will be passed during execution to the stage-level input(s) that link to it.
Defining inputs
for the workflow creates a special type of a workflow called locked workflow. Locked workflows are workflows in which certain input fields cannot be overriden when the workflow is initialized to run. This is achieved by the inputs
property, which acts as an allowable list for those inputs which are "unlocked". If the workflow creator defines this property, the inputs listed in this array can be set by the user when they run the workflow (they are considered "unlocked"), and all other inputs are automatically "locked". When the inputs
property is undefined or null the workflow is fully unlocked and acts like any other regular workflow where all the inputs can be provided or overriden by the user that runs the workflow. When the inputs
property is set to an empty array, there are no unlocked fields so the workflow is fully locked.
Workflow Output
You can also use stage references as values to link an input to the input or output of another stage. These references are hashes with a single key $dnanexus_link
whose value is a hash with exactly two keys/values:
stage
string another stage's ID whose output will be used
exactly one of the following key/values:
outputField
string the output field name of the stage's executable to be used
inputField
string the input field name of the stage's executable to be used
and, optionally:
index
integer the index of the array that is the output or input of the linked stage; this is 0-indexed, so a value of 0 indicates the first element should be used.
If the workflow has defined inputs
, you can use workflow input references to link stage inputs to the workflow level inputs. These references are hashes with a singe key $dnanexus_link
whose value is a hash with exactly one key/value:
workflowInputField
: string the input field name of the current workflow
Linking input to other stage output
Using the outputField
option is useful for chaining the output of a stage to the input of another stage to make an analysis pipeline. For example, a first stage (stage-xxxx
) could maps reads to a reference genome and then pass those mappings on to a second stage (stage-yyyy
) that will call variants on those mappings. We can do this by setting the following input for the second stage:
When the workflow is run, the second stage will receive the mappings input once the first stage has finished.
Linking input to other stage input
Linking input fields together can also be useful. For example, if there are two stages which require the same reference genome, we can link the input of one (stage-xxxx
) to the other (stage-yyyy
) by setting the input of the first as follows:
When running the workflow, the reference genome input only needs to be provided once to the input of stage-yyyy
, and the other stage stage-xxxx
will inherit the same value.
Linking workflow input to stage input
It is possible to link stage input to the input of the current workflow. For example, if the stage-xxxx
requires a reference genome, we can link the input of stage-xxxx
to the input of the workflow as follows:
The workflow inputs
field should then be defined for the workflow, for example:
During runtime stage inputs will then consume the input values provided on the workflow level, i.e. the value passed to the field reference_genome
will be used by reference_genome_field_of_stage_xxxx
.
Note that hiding an output for a stage has further consequences; the output is treated as intermediate output and is deleted after the analysis has finished running.
When specifying input for /workflow-xxxx/run, the input field names for an analysis are automatically generated to have the form "<stage ID>.<input field name>" if the input is provided to a stage directly, or "<input field name>" if it is the input defined for the workflow.
Connecting the input to the input or output of another stage in the workflow is also possible. In such a situation, a workflow stage reference should be used. To reference the input of another stage, say of stage "stage-xxxx" with input "reference_genome", you would provide the value:
When the workflow is run, this will be translated into whatever value is given as input for "reference_genome" for the stage "stage-xxxx" in the workflow.
If the key outputField
is used in place of inputField
, then the value represents the output of that stage instead. When the workflow is run and an analysis created, the workflow stage reference will be translated into an analysis stage reference:
which will be resolved when the stage "stage-xxxx" finishes running in analysis "analysis-xxxx".
/workflow/new
Specification
Creates a new workflow data object which can be used to execute a series of apps, applets, and/or workflows.
Inputs
project
string ID of the project or container to which the workflow should belong (i.e. the string "project-xxxx")
name
string (optional, default is the new ID) The name of the object
title
string or null (optional, default null) Title of the workflow, e.g. “Micro Map Pipeline”; if null, then the name of the workflow will be used as the title
summary
string (optional, default "") A short description of the workflow
description
string (optional, default "") A longer description about the workflow
tags
array of strings (optional) Tags to associate with the object
types
array of strings (optional) Types to associate with the object
hidden
boolean (optional, default false) Whether the object should be hidden
properties
mapping (optional) Properties to associate with the object
key Property name
value string Property value
folder
string (optional, default "/") Full path of the folder that is to contain the new object
parents
boolean (optional, default false) Whether all folders in the path provided in folder
should be created if they do not exist
initializeFrom
mapping (optional) Indicate an existing workflow or analysis from which to use the metadata as default values for all fields that are not given:
id
string ID of the workflow or analysis from which to retrieve workflow metadata
project
string (required for workflow IDs and ignored otherwise) ID of the project in which the workflow specified in id
should be found
stages
array of mappings (optional) Stages to add to the workflow. If not supplied, the workflow that is created will be empty. Each value is a mapping with the key/values:
executable
string ID of app or applet to be run in this stage
name
string (optional) Name (display label) for the stage
key Input field name
value Input field value
restartOn
mapping (optional) Indicate a job restart policy
key A restartable failure reason ("ExecutionError", "UnresponsiveWorker", "JMInternalError", or "AppInternalError") or "*" to indicate all restartable failure reasons that are otherwise not present as keys
value int Maximum number of restarts for the failure reason
maxRestarts
int (optional, default 9) Non-negative integer less than 10, indicating the maximum number of times that the job will be restarted
onNonRestartableFailure
string (optional, default "failStage") Either the value "failStage" or "failAllStages"; indicates whether the failure of this stage (when run as part of an analysis) should force all other non-terminal stages in the analysis to fail as well if a non-restartable failure occurs, even if those stages do not have any dependencies on this stage. (Stages that have dependencies on this stage will still fail irrespective of this setting.)
ignoreReuse
array of strings (optional) Specifies ids of workflow stages (or "*" for all stages) that will ignore job reuse. If a specified stage points to a nested sub-workflow, reuse will be ignored recursively by the whole nested sub-workflow. Overrides ignoreReuse setting in stage executables.
Outputs
id
string ID of the created workflow object (i.e. a string in the form "workflow-xxxx")
editVersion
int The initial edit version number of the workflow object
Errors
InvalidInput
A reserved linking string ("$dnanexus_link") appears as a key in a hash in "details" but is not the only key in the hash
A reserved linking string ("$dnanexus_link") appears as the only key in a hash in details
but has value other than a string
The id
given under initializeFrom
is not a valid workflow or analysis ID
"project" is missing if id
given under initializeFrom
is a workflow ID
For each property key-value pair, the size, encoded in UTF-8, of the property key may not exceed 100 bytes and the property value may not exceed 700 bytes
A nonce
was reused in a request but some of the other inputs had changed signifying a new and different request
A nonce
may not exceed 128 bytes
InvalidType
The project
is not a valid project ID
PermissionDenied
CONTRIBUTE access required, VIEW access required for the project specified under initializeFrom
if a workflow or analysis was specified.
ResourceNotFound
The specified project is not found
The path in folder
does not exist while parents
is false, or a specified project, workflow, or analysis ID specified in initializeFrom
is not found, or a stage in ignoreReuse
is not found)
/workflow-xxxx/overwrite
Specification
Overwrites the workflow with the workflow-specific metadata from another workflow or an analysis other than the editVersion. The workflow's name, tags, properties, types, visibility, and details are left unchanged.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
from
mapping Indicate the existing workflow or analysis from which to use the metadata
id
string ID of the workflow or analysis from which to retrieve workflow metadata
project
string (required for workflow IDs and ignored otherwise) ID of the project ID in which the workflow specified in id
should be found
Outputs
id
string ID of the manipulated workflow
editVersion
int The new edit version number
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
from
is not a hash
from.id
is not a string
from.project
is not a string if from.id
is a workflow ID
ResourceNotFound
The specified workflow does not exist
The workflow or analysis specified in from
cannot be found
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
User does not have VIEW access to the project containing the workflow or analysis represented in from
/workflow-xxxx/addStage
Specification
Adds a stage to the workflow.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
executable
string App or applet ID
name
string or null (optional, default null) Name (display label) for the stage, or null to indicate no name
restartOn
mapping (optional) Indicate a job restart policy
key A restartable failure reason ("ExecutionError", "UnresponsiveWorker", "JMInternalError", or "AppInternalError") or "*" to indicate all restartable failure reasons that are otherwise not present as keys
value int Maximum number of restarts for the failure reason
maxRestarts
int (optional, default 9) Non-negative integer less than 10, indicating the maximum number of times that the job will be restarted
onNonRestartableFailure
string (optional, default "failStage") Either the value "failStage" or "failAllStages"; indicates whether the failure of this stage (when run as part of an analysis) should force all other non-terminal stages in the analysis to fail as well if a non-restartable failure occurs, even if those stages do not have any dependencies on this stage. (Stages that have dependencies on this stage will still fail irrespective of this setting.)
Outputs
id
string ID of the manipulated workflow
editVersion
int The new edit version number
stage
string ID of the new stage
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
executable
is not a string
name
if provided is not a string
folder
if provided is not a valid folder path
input
if provided is not a hash or is not valid input for the specified executable
executionPolicy
if provided is not a hash
executionPolicy.restartOn
if provided is not a hash, contains a failure reason key that cannot be restarted, or contains a value which is not an integer between 0 and 9
executionPolicy.onNonRestartableFailure
is not one of the allowed values
ResourceNotFound
The specified workflow does not exist
The specified executable does not exist
A provided input value in input
could not be found
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
An accessible copy of the executable could not be found
/workflow-xxxx/removeStage
Specification
Removes a stage from the workflow.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
stage
string ID of the stage to remove
Outputs
id
string ID of the manipulated workflow
editVersion
int The new edit version number
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
stage
is not a string
ResourceNotFound
The specified workflow does not exist
The specified stage does not exist in the workflow
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
/workflow-xxxx/moveStage
Specification
Reorders the stages by moving a specified stage to a new index or position in the workflow. This does not affect how the stages are run but is merely for personal preference and organization.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
stage
string ID of the stage to move
newIndex
int The index or key that the stage will have after the move; all other stages will be moved to accommodate this change; must be in [0, n), where n is the total number of stages
Outputs
id
string ID of the manipulated workflow
editVersion
int The new edit version number
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
stage
is not a string
newIndex
is not in the range [0, n) where is the number of stages in the workflow
ResourceNotFound
The specified workflow does not exist
The specified stage does not exist in the workflow)
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
/workflow-xxxx/update
Specification
Update the workflow with any fields that are provided.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
title
string or null (optional) The workflow’s title, e.g. "Micro Map Pipeline"; if null, the name of the workflow will be used as the title
summary
string (optional) A short description of the workflow
description
string (optional) A longer description about the workflow
stages
mapping (optional) Updates for one or more of the workflow's stages
key ID of the stage to update
value mapping Updates to make to the stage
name
string or null (optional) New name for the stage; use null to unset the name
executionPolicy
mapping (optional) Set the default execution policy for this stage; use the empty mapping { } to unset
inputSpecMods
mapping (optional) Update(s) to how the stage input specification is exported for the workflow; any subset can be provided
key Input field name from this stage's executable
value mapping Updates for the specified stage input field name
name
string or null (optional) The canonical name by which a stage's input can be addressed when running the workflow is of the form "<stage ID>.<original field name>". By providing a different string here, you will override the name as shown in the inputSpec
of the workflow, and it can be used when giving input to run the workflow. Note that the canonical name value can still be used to refer to this input, but both names cannot be used at the same time. If null
is provided, then any previously-set name will be dropped, and only the canonical name can be used.
label
string or null (optional) A replacement label for the input parameter. If null is provided, then any previously-set label will be dropped, and the original executable's label will be used.
help
string or null (optional) A replacement help string for the input parameter. If null is provided, then any previously-set help string will be dropped, and the original executable's help string will be used.
group
string or null (optional) A replacement group for the input parameter. The default group for a stage's input is the stage's ID (if it had no group in the executable), or the string "<stage ID>:<group name>" (if it was part of a group in the executable). By providing a different string here, you override the group in which the input parameter appears in the inputSpec
of the workflow. If the null value is provided, then any previously-set group value will be dropped, and the canonical group name will be used. If the empty string is provided, the parameter will not be in any group.
hidden
boolean (optional) Whether to hide the input parameter from the inputSpec
of the workflow; note that the input can still be provided and overridden by its name "<stage ID>.<original field name>".
outputSpecMods
mapping (optional) Update(s) to how the stage output specification is exported for the workflow; any subset can be provided. This field follows the same syntax as for inputSpecMods
defined above and behaves roughly the same but modifies outputSpec
instead. The exception in behavior occurs for the hidden
field. If an output has hidden
set to true
, its data object value (if applicable) will not be cloned into the parent container when the stage or analysis is done. This may be a useful feature if a stage in your analysis produces many intermediate outputs that are not relevant to the analysis or are not ultimately useful once the analysis has finished.
Outputs
id
string ID of the manipulated workflow
editVersion
int The new edit version number
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
title
if provided is not a string nor null
summary
if provided is not a string
description
if provided is not a string
stages
if provided is not a hash
A key in stages
is not a stage ID string
name
if provided in a stage hash is not a string
folder
if provided in a stage hash is not a valid folder path
input
if provided in a stage hash is not a hash or is not valid input for the specified executable
inputSpecMods
or outputSpecMods
if provided in a stage hash is not a hash or contains a key which does not abide by the syntax specification above
ResourceNotFound
The specified workflow does not exist
One of the specified stage IDs could not be found in the workflow
A provided input value in an input
hash in a stage's hash could not be found
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
/workflow-xxxx/isStageCompatible
Specification
Check whether the proposed replacement executable for a stage is going to be a fully compatible replacement or not.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
stage
string ID of the stage to check for compatibility
executable
string ID of the executable that would be used as a replacement
Outputs
id
string ID of the workflow that was checked for compatibility
compatible
boolean The value true if it is compatible and false otherwise
If compatible
is false, the following key is also present:
incompatibilities
array of strings A list of reasons for which the two executables are not compatible
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
stage
is not a string
executable
is not a string
The given executable is missing an input or output specification
ResourceNotFound
The specified workflow does not exist
The specified stage does not exist in the workflow
The specified executable does not exist
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
PermissionDenied
User does not have VIEW access to the workflow's project required
An accessible copy of the executable could not be found
/workflow-xxxx/updateStageExecutable
Specification
Update the executable to be run in one of the workflow's stages.
Inputs
editVersion
int The edit version number that was last observed, either via /workflow-xxxx/describe
or as output from an API call that changed the workflow; this value must match the current version stored in the workflow object for the API call to succeed
stage
string ID of the stage to update with the executable
executable
string ID of the executable to use for the stage
force
boolean (optional, default false) Whether to update the executable even if the one specified in executable
is incompatible with the one that is currently used for the stage
Outputs
id
string ID of the workflow
editVersion
int The new edit version number
compatible
boolean Whether executable
was compatible; if false, then further action (such as setting new inputs) may need to be taken in order to run the workflow as is
If compatible
is false, the following is also present:
incompatibilities
list of strings A list of reasons for which the two executables are not compatible
Errors
InvalidInput
Input is not a hash
editVersion
is not an integer
stage
is not a string, executable
is not a string
The given executable is missing an input or output specification
force
is not a boolean
ResourceNotFound
The specified workflow does not exist
The specified stage does not exist in the workflow
The specified executable does not exist
InvalidState
Workflow is not in the "open" state
editVersion
provided does not match the current stored value
The requested executable is not compatible with the previous executable
force
was not set to true
PermissionDenied
User does not have CONTRIBUTE access to the workflow's project
An accessible copy of the executable could not be found
/workflow-xxxx/describe
Specification
Describes the specified workflow object.
Inputs
project
string (optional) Project or container ID to be used as a hint for finding an accessible copy of the object
defaultFields
boolean (optional, default false if fields
is supplied, true otherwise) whether to include the default set of fields in the output (the default fields are described in the "Outputs" section below). The selections are overridden by any fields explicitly named in fields
.
fields
mapping (optional) include or exclude the specified fields from the output. These selections override the settings in defaultFields
.
key Desired output field; see the "Outputs" section below for valid values here
value boolean whether to include the field
includeHidden
boolean (optional; default false) Whether hidden input and output parameters should appear in the inputSpec
and outputSpec
fields
getRerunInfo
boolean (optional, default false) Whether rerun information should be returned for each stage
rerunStages
array of strings (optional) Applicable only if getRerunInfo
is set to true; a set of stage IDs that would be forcibly rerun and to return rerun information accordingly
rerunProject
string (optional, default is the value of project
returned) Project ID to use for retrieving rerun information
The following options are deprecated (and will not be respected if fields
is present):
properties
boolean (optional, default false) Whether the properties should be returned
details
boolean (optional, default false) Whether the details should also be returned
Outputs
id
string The object ID (i.e. the string "workflow-xxxx")
The following fields are included by default (but can be disabled using fields
or defaultFields
):
project
string ID of the project or container in which the object was found
class
string The value "workflow"
types
array of strings Types associated with the object
created
timestamp Time at which this object was created
state
string Either "open" or "closed"
hidden
boolean Whether the object is hidden or not
links
array of strings The object IDs that are pointed to from this object
name
string The name of the object
folder
string The full path to the folder containing the object
sponsored
boolean Whether the object is sponsored by DNAnexus
tags
array of strings Tags associated with the object
modified
timestamp Time at which the user-provided metadata of the object was last modified
createdBy
mapping How the object was created
user
string ID of the user who created the object or launched an execution which created the object
job
string present if a job created the object ID of the job that created the object
executable
string present if a job created the object ID of the app or applet that the job was running
title
string The workflow's effective title (will always equal name
if it has not been set to a string)
summary
string The workflow's summary
description
string The workflow's description
inputs
array of mappings, or null Input specification of the workflow (not the input of particular stages, which is returned in inputSpec
)
outputs
array of mappings, or null Output specification of the workflow (not the output of stages, which is returned in outputSpec
)
editVersion
int The current edit version of the workflow; this value must be provided with any of the workflow-editing API methods to ensure that simultaneous edits are not occurring
ignoreReuse
array of strings, or null workflow stage ids that are configured to ignore job reuse
stages
array of mappings List of metadata for each stage; each value is a mapping with the key/values:
id
string Stage ID
executable
string App or applet ID
name
string or null Name of the stage or null if not set
input
mapping Input (possibly partial) to the stage's executable that has been bound
accessible
boolean Whether the executable is accessible
executionPolicy
mapping The default execution policy for this stage
systemRequirements
mapping The requested systemRequirements
value for the stage
inputSpecMods
mapping Modifications for the stage's input parameters when represented in the workflow's input specification
key Input parameter name from this stage's executable
value mapping Modifications for the input parameter
name
string (present if set) Replacement name of the input parameter; this is guaranteed to be unique in the stages input specification
label
string (present if set) Replacement label for the input parameter
help
string (present if set) Replacement help string for the input parameter
group
string The group to which the input parameter belongs (the empty string indicates no group)
hidden
boolean (present if true) Whether the input field is hidden from the workflow's input specification
outputSpecMods
mapping Modifications for restricting the stages' output and representing its output
key Output parameter name from this stage's executable
value mapping Modifications for the output parameter with any number of the same key/values that are also present in inputSpecMods
. Note that if an output has hidden
set to true
, its data object value (if applicable) will not be cloned into the parent container when the stage or analysis is done and will be deleted immediately upon completion or failure of the analysis if delayWorkspaceDestruction
is not set to true. If getRerunInfo
is true, the following keys are present:
wouldBeRerun
boolean Whether the stage would be rerun if the workflow were to be run (taking into account the value given for rerunStages
, if applicable)
cachedExecution
string (present if wouldBeRerun
is false) The job ID from which the outputs would be used
cachedOutput
mapping or null (present if wouldBeRerun
is false) The output from the cached execution if available or null if the execution has not finished yet
initializedFrom
mapping (present if the workflow was created using the initializedFrom
option) Basic metadata recording how this workflow was created
editVersion
int (present if id
is a workflow ID) The editVersion of the original workflow at the time of creation
The following field (included by default) is available if the object is sponsored by a third party:
sponsoredUntil
timestamp Indicates the expiration time of data sponsorship (this field is only set if the object is currently sponsored, and if set, the specified time is always in the future)
The following fields are only returned if the corresponding field in the fields
input is set to true
:
properties
mapping Properties associated with the object
key Property name
value string Property value
details
mapping or array Contents of the object’s details
Errors
ResourceNotFound
The specified object does not exist or the specified project does not exist
InvalidInput
The input is not a hash
project
(if present) is not a string
The value of properties
(if present) is not a boolean
includeHidden
if present is not a boolean
getRerunInfo
if present is not a boolean
rerunStages
if present is not an array of nonempty strings
InvalidType
rerunProject
(if present) is not a project ID
PermissionDenied
VIEW access is required for the project
input if provided
VIEW access is required for some project containing the specified object, may be different from the project
input provided.
/workflow-xxxx/run
Specification
All inputs must be provided, either as bound inputs in the workflow or separately in the input
field.
Intermediate results will be output for the stages and outputs specified.
If any stages have been previously run with the same executable and the same inputs, then the previous results may be used.
Inputs
name
string (optional, default is the workflow name) Name for the resulting analysis
input
mapping Input for the analysis is launched with
key Input field name; see the inputSpec
and inputs
fields in the output of /workflow-xxxx/describe
for what the names of the inputs are
value Input field value
key Stage ID or "*" to indicate that the value should be applied to all stages not otherwise mentioned
value null or string Value to replace the stored default
details
array or mapping (optional, default { }) Any conformant JSON (i.e. a JSON object or array, per RFC4627), which will be stored with the created job
delayWorkspaceDestruction
boolean (optional) If not given, the value defaults to false for root executions (launched by a user or detached from another job), or to the parent's delayWorkspaceDestruction
setting. If set to true, the temporary workspace created for the resulting execution will be preserved for 3 days after the job either succeeds or fails.
rerunStages
array of strings (optional) A list of stage IDs that should be forcibly rerun (which will be rerun in addition to other stages that the system will identify as requiring a rerun as well); if the list includes the string "*", then all stages will be rerun
restartOn
mapping (optional) Indicate a job restart policy
key A restartable failure reason ("ExecutionError", "UnresponsiveWorker", "JMInternalError", or "AppInternalError") or "*" to indicate all restartable failure reasons that are otherwise not present as keys
value int Maximum number of restarts for the failure reason
maxRestarts
int (optional, default 9) Non-negative integer less than 10, indicating the maximum number of times that the job will be restarted
onNonRestartableFailure
string (optional) If unset, allows the stages to govern their failure propagation behavior. If set, must be either the value "failStage" or "failAllStages", indicating whether the failure of any stage should propagate failure to all other non-terminal stages in the analysis, even if those stages do not have any dependencies on the failed stage. (Stages that have dependencies on the stage that failed will still fail irrespective of this setting.)
key Stage ID
value mapping Value to override or merge with the stage's systemRequirements
value
allowSSH
array of strings (optional, default [ ]) Array of IP addresses or CIDR blocks (up to /16) from which SSH access will be allowed to the user by the worker running this job. Array may also include '*' which is interpreted as the IP address of the client issuing this API call as seen by the API server.
debug
mapping (optional, default { }) Specify debugging options for running the executable; this field is only accepted when this call is made by a user (and not a job)
debugOn
array of strings (optional, default [ ]) Array of job errors after which the job's worker should be kept running for debugging purposes, offering a chance to SSH into the worker before worker termination (assuming SSH has been enabled). This option applies to all jobs in the execution tree. Jobs in this state for longer than 2 days will be automatically terminated but can be terminated earlier. Allowed entries include "ExecutionError", "AppError", and "AppInternalError".
editVersion
int (optional) If provided, run the workflow only if the current version matches the provided value and throw an error if it does not match; if not provided, the current version is run.
properties
mapping (optional) Properties to associate with the resulting analysis.
key Property name
value string Property value
tags
array of strings (optional) Tags to associate with the resulting analysis.
singleContext
boolean (optional) If true then the resulting jobs and all of their descendants will only be allowed to use the authentication token given to them at the onset. Use of any other authentication token will result in an error. This option offers extra security to ensure data cannot leak out of your given context. In restricted projects user-specified value is ignored, and singleContext: true
setting is used instead.
ignoreReuse
array of strings (optional) Specifies ids of workflow stages (or "*" for all stages) that will ignore job reuse. If a specified stage points to a nested sub-workflow, reuse will be ignored recursively by the whole nested sub-workflow. Overrides ignoreReuse setting in the workflow and in stage executables.
detach
boolean (optional) This option has no impact when the API is invoked by a user. If invoked from a job with detach set to true, the new analysis will be detached from the creator job and will appear as a typical root execution. A failure in the detached analysis will not cause a termination of the job from which it was created and vice versa. Detached job will inherit neither the access to the workspace of its creator job nor the creator job's priority. Detached analysis' access permissions will be the intersection (most restricted) of access permissions of the creator job and the permissions requested by jobs' executables in the detached analysis. To launch the detached analysis, creator job must have CONTRIBUTE or higher access to the project in which the detached job is launched. Additionally, the billTo of the project in which the creator job is running must be licensed to launch detached executions.
rank
integer (optional) An integer between -1024 and 1023, inclusive. The rank indicates the priority in which the executions generated from this executable will be processed. The higher the rank, the more prioritized it will be. If no rank is provided, the executions default to a rank of zero. If the execution is not a root execution, it will inherit its parent's rank. If a rank is provided, all executions relating to the workflow stages will also inherit the rank.
costLimit
float (optional) The limit of the cost that this execution tree should accrue before termination. This field will be ignored if this is not a root execution.
preserveJobOutputs
mapping (optional, default is null). Preserves all cloneable outputs of every completed, non-jobReused job in the execution tree launched by this api call in the root execution project, even if root execution ends up failing. Preserving the job outputs in the project trades off higher costs of storage for the possibility of subsequent job reuse.
When a non-jobReused job in the root execution tree launched with non-null preserveJobOutputs
enters "done" state, all cloneable objects (e.g. files, records, applets, closed workflows, but not databases) referenced by the $dnanexus_link
in the job's output
field will be cloned to the project folder described by preserveJobOutputs.folder
, unless the output objects already appear elsewhere in the project. If the folder specified by preserveJobOutputs.folder
does not exist in the project, the system will create the folder and its parents.
As the root job or root analysis' stages complete, the regular outputs of the root execution will be moved from preserveJobOutputs.folder
to the regular output folder(s) of the root execution. So if you [1] run your root execution without the preserveJobOutputs
option to completion, some root execution outputs will appear in the project in the root execution's output folder(s). If you had run the same execution with preserveJobOutputs.folder
set to "/pjo_folder"
, the same set of outputs would appear in the same set of root execution folders as in [1] at completion of the root execution, while some additional job outputs that are not outputs of the root execution would appear in "/pjo_folder"
.
preserveJobOutputs
argument can be specified only when starting a root execution or a detached job.
preserveJobOutputs
value, if not null, should be a mapping that may contain the following:
key "folder"
string (optional)
valuepath_to_folder
string (required if "folder" key is specified). Specifies a folder in the root execution project where the outputs of jobs that are a part of the launched execution will be stored. path_to_folder
starting with "/
" is interpreted as absolute folder path in the project the job is running in. path_to_folder
not starting with "/
" is interpreted as a path relative to the root execution's folder
field. path_to_folder
value of ""
(i.e. empty string) will preserve job outputs in the folder described by root execution's folder
field.
If the preserveJobOutputs
mapping does not have a folder
key, the system will use the default folder value of "intermediateJobOutputs"
(i.e. "preserveJobOutputs": {}
is equivalent to
"preserveJobOutputs": {"folder":"intermediateJobOutputs"}
).
It is recommended to place preserveJobOutputs outputs for different root executions into different folders so as not to create a single folder with a very large (>450K) number of files.
Outputs
id
string ID of the created analysis object (i.e. a string in the form “analysis-xxxx”).
stages
array of strings List of job IDs that will be created for each stage, as ordered in the workflow.
Errors
ResourceNotFound
The specified workflow object, any referenced apps or applets, or project context does not exist
PermissionDenied
VIEW access to the workflow, VIEW access to applets, any apps must be installed
CONTRIBUTE access to the project context required unless called by a job
When specifying allowSSH
or debug
options, the user must have developer access to all apps in the workflow, or the apps must have the openSource
field set to true
If preserveJobOutputs
is not null and billTo
of the project where execution is attempted does not have preserveJobOutputs license.
detailedJobMetrics
setting of true requires project's billTo
to have detailedJobMetrics
license feature set to true.
app{let}-xxxx
can not run in project-xxxx
because executable's httpsApp.shared_access
should be NONE
to run with isolated browsing.
Note that this check is applied to all of workflow's stages.
InvalidInput
The workflow spec is not complete
The project context must be in the same region as this workflow
All data object inputs that are specified directly must be in the same region as the project context.
All inputs that are job-based object references must refer to a job that was run in the same region as the project context.
allowSSH
accepts only IP addresses or CIDR blocks up to /16
A nonce
was reused in a request but some of the other inputs had changed signifying a new and different request
A nonce
may not exceed 128 bytes
The billTo
of the job's project must be licensed to start detached executions when invoked from the job with detach: true
argument
preserveJobOutputs
is specified when launching a non-detached execution from a job.
preserveJobOutputs.folder
value is a syntactically invalid path to a folder.
detailedJobMetrics
can not be specified when launching a non-detached execution from a job.
timeoutPolicyByExecutable
for all executables should not be null
timeoutPolicyByExecutable
for all entry points of all executables should not be null
timeoutPolicyByExecutable
for all entry points of all executables should not exceed 30 days
Expected key "timeoutPolicyByExecutable
.*" of input to match "/^(app|applet)-[0-9A-Za-z]{24}$/
InvalidState
editVersion
was provided and does not match the current sorted value)
/workflow-xxxx/dryRun
Specification
No new jobs or analyses are created as a result of this method. Any analysis and job IDs returned in the response (with the exception of cached execution IDs) are placeholders and do not represent actual entities in the system.
Note that this method can be used to determine which stages have previous results that would be used. In particular, a stage that would reuse a cached result will have a parentAnalysis
field (found at stages.N.execution.parentAnalysis
where N is the index of the stage) that will refer to a preexisting analysis and will therefore not match the top-level field id
in the response.
Inputs
Outputs
Errors
/workflow-xxxx/validateBatch
Specification
This API call verifies that a set of input values for a particular workflow can be used to launch a batch of jobs in parallel.
Batch and common inputs:
batchInput
: mapping of inputs corresponding to batches. The nth value of each array corresponds to nth execution of the workflow. Including a null
value in an array at a given position means that the corresponding workflow input field is optional and the default value, if defined, should be used. E.g.:
commonInput
: mapping of non-batch, constant inputs common to all batch jobs, e.g.:
File references:
files
: list of files (passed as $dnanexus_link references), must be a superset of files included in batchInput
and/or commonInput
e.g.:
Output: list of mappings, each mapping corresponds to an expanded batch call. Nth mapping contains the input values with which the th execution of the workflow will be run, e.g.:
It performs the following validation:
the input types match the expected workflow input field types,
provided inputs are sufficient to run the workflow,
null values are only among values for inputs that are optional or
have no specified default values,
all arrays of batchInput
are of equal size,
every file referred to in batchInputs
exists in files
input.
If the workflow is locked, i.e. workflow-level inputs
are specified for the workflow, this inputs
specification will be used in place of stage-level inputSpecs
, and workflow input field names must be provided in batchInput
and commonInput
. The reason for this that for locked workflows input values can only be passed to the workflow-level inputs. Thus, for locked workflow we should refer to input fields by their names defined in inputs
. In order to refer to a specific field in a stage of a non-locked workflow, the <stage id>
.<input field name defined in inputSpec>
format should be used.
Inputs
batchInput
mapping Input that the workflow is launched with
key Input field name. It must be one of the names of the inputs defined in the workflow input specification.
value Input field values. It must be an array of fields.
commonInput
mapping (optional) Input that the workflow is launched with
key Input field name. It must be one of the names of the inputs defined in the workflow input specification.
value Input field values. It must be an array of fields.
files
list (optional) Files that are needed to run the batch jobs, they must be provided as $dnanexus_links
. They must correspond to all the files included in commonInput
or batchInput
.
Outputs
expandedBatch
list of mappings Each mapping contains the input values for one execution of the workflow in batch mode.
Errors
InvalidInput
Input specification must be specified for the workflow
Expected batchInput
to be a JSON object
Expected commonInput
to be a JSON object
Expected files
to be an array of $dnanexus_link
references to files
The batchInput
field is required but empty array was provided
Expected the value of batchInput
for an workflow input field to be an array
Expected the length of all arrays in batchInput
to be equal
The workflow input field value must be specified in batchInput
The workflow input field is not defined in the input specification of the workflow
All the values of a specific batchInput
field must be provided (cannot be null
) since the field is required and has no default value
Expected all the files in batchInput
and commonInput
to be referenced in the files
input array
/analysis-xxxx/describe
Specification
Describe the specified analysis object.
If the results from previously run jobs are used for any of the stages, they will still be listed here. Note, however, that the stages' parentAnalysis
field will still reflect the original analysis(es) in which they were run.
The description of an analysis may not be available if an upstream analysis is not finished running. Users with reorg apps that rely on describing the currently running analysis may want to check the output field "dependsOn" before the full analysis description becomes available using dx describe analysis-xxx --json | jq -r .dependsOn
or equivalent dxpy
bindings. The output of the command will be an empty array "[]" if it no longer depends on anything (e.g. status "done") which is the signal to proceed, or some analysis / subanalysis IDs if it is not ready, and the reorg script should wait.
Inputs
defaultFields
boolean (optional, default false if fields
is supplied, true otherwise) Specifies whether to include the default set of fields in the output (the default fields are described in the "Outputs" section below). The selections are overridden by any fields explicitly named in fields.
fields
: mapping (optional) include or exclude the specified fields from the output. These selections override the settings in defaultFields
.
key
string Desired output field; see the "Outputs" section below for valid values here
value
boolean Whether to include the field
id
string The object ID (i.e. the string "analysis-xxxx")
The following fields are included by default (but can be disabled by setting defaultFields
to false
or by using the fields
) input:
class
string The value "analysis"
name
string Name of the analysis (either specified at creation time or given automatically by the system)
executable
string ID of the workflow or the global workflow that was run
executableName
string Name of the the workflow or the global workflow that was run
created
timestamp Time at which this object was created
modified
timestamp Time at this analysis was last updated
billTo
string ID of the account to which any costs associated with this analysis will be billed
project
string ID of the project in which this analysis was run
folder
string The output folder in which the outputs of this analysis will be placed
rootExecution
string ID of the job or analysis at the root of the execution tree (the job or analysis created by a user's API call rather than called by a job or as a stage in an analysis)
parentJob
string or null ID of the job which created this analysis, or null if this analysis was not created by a job
parentJobTry
non-negative integer or null. null
is returned if this analysis was not created by a job, or if the parent job had a null try
attribute. Otherwise, this analysis was created from the parentJobTry
try of the parentJob
.
parentAnalysis
string or null If this is an analysis that was run as a stage in an analysis, then this is the ID of that analysis; otherwise, it is null
detachedFrom
string or null The ID of the job this analysis was detached from via the detach
option, otherwise null
detachedFromTry
non-negative integer or null. If this analysis was detached from a job, detachedFrom
and detachedFromTry
describe the specific try of the job this analysis was detached from. null
is returned if this analysis was not detached from another job or if the detachedFrom
had a null
try
attribute.
analysis
string or null Null if this analysis was not run as part of a stage in an analysis; otherwise, the ID of the analysis this analysis is part of
stage
string or null Null if this job was not run as part of a stage in an analysis; otherwise, the ID of the stage this analysis is part of
workflow
mapping Metadata of the workflow that was run, including at least the following fields (newer analyses created after 8/2014 will include the full describe output at the time that the analysis was created):
id
string ID of the workflow
name
string Name of the workflow
inputs
array of mappings Input specification of the workflow
outputs
array of mappings Output specification of the workflow
editVersion
int Edit version at the time of running the workflow
initializedFrom
mapping If applicable, the initializedFrom
mapping from the workflow
stages
array of mappings List of metadata for each of the stages' executions
id
string Stage ID
execution
mapping fixme with key id
and value of the execution ID; additional keys are present if the describe hash of the origin job or analysis of the stage has been requested and is available (the fields returned here can be limited by setting fields.stages
in the input to the hash one would give to describe the execution)
state
string The analysis state, one of "in_progress", "partially_failed", "done", "failed", "terminating", and "terminated"
workspace
string ID of the temporary workspace assigned to the analysis (e.g. “container-xxxx”)
launchedBy
string ID of the user who launched rootExecution
; this is propagated to all jobs launched by the analysis
tags
array of strings Tags associated with the analysis
properties
mapping Properties associated with the analysis
key Property name
value string Property value
details
array or mapping The JSON details that were stored with this analysis
runInput
mapping The value given as input
in the API call to run the workflow
originalInput
mapping The effective input of the analysis, including all defaults as bound in the stages of the workflow, overridden with any values present in runInput
, and all input field names are translated to their canonical names, i.e. of the form "<stage ID>.<field name>"
input
mapping The same as originalInput
output
mapping or null Null if no stages have finished; otherwise, contains key/value pairs for all outputs that are currently available (final only when state
is one of "done", "terminated", and "failed")
delayWorkspaceDestruction
boolean Whether the analysis's temporary workspace will be kept around for 3 days after the analysis either succeeds or fails
ignoreReuse
array of strings, or null analysis stage ids (or "*" for all stages) that were configured to ignore job reuse.
preserveJobOutputs
null or a mapping with preserveJobOutputs.folder
expanded to start with "/"
.
detailedJobMetrics
boolean Set to true only if the detailed job metrics collection was enabled for this analysis.
costLimit
float If the job is a root execution, and has the root execution cost limit, this is the cost limit for the root execution.
rank
int The rank of the analysis, with a range from [-1024 to 1023].
If this job is a root execution, the following fields are included by default (but can be disabled using fields
):
selectedTreeTurnaroundTimeThreshold
integer or null The selected turnaround time threshold (in seconds) for this root execution. When treeTurnaroundTime
reaches the selectedTreeTurnaroundTimeThreshold
, the system sends an email about this root execution to the launchedBy
user and the billTo
profile.
selectedTreeTurnaroundTimeThresholdFrom
string or null Where selectedTreeTurnaroundTimeThreshold
is from. executable
means that selectedTreeTurnaroundTimeThreshold
is from this root execution's executable's treeTurnaroundTimeThreshold
. system
means that selectedTreeTurnaroundTimeThreshold
is from the system's default threshold.
If the requesting user has permissions to view the pricing model of the billTo
of the analysis, and the price for the analysis has been finalized:
currency
mapping Information about currency settings, such as dxCode, code, symbol, symbolPosition, decimalSymbol and groupingSymbol.
totalPrice
number Price (in currency
) for how much this job (along with all its subjobs) costs.
priceComputedAt
timestamp Time at which totalPrice
was computed. For billing purposes, the cost of the analysis accrues to the invoice of the month that contains priceComputedAt
(in UTC).
totalEgress
mapping Egress (in Byte
) for how much data amount this job (along with all its subjobs) has egressed.
regionLocalEgress
int Amount in bytes of data transfer between IP in the same cloud region.
internetEgress
int Amount in bytes of data transfer to IP outside of the cloud provider.
interRegionEgress
int Amount in bytes of data transfer to IP in other regions of the cloud provider.
egressComputedAt
timestamp Time at which totalEgress
was computed. For billing purposes, the cost of the analysis accrues to the invoice of the month that contains egressComputedAt (in UTC).
The following field is only returned if the corresponding field in the fields
input is set to true
, the requesting user has permissions to view the pricing model of the billTo
of the job, and the job is a root execution:
subtotalPriceInfo
mapping Information about the current costs associated with all jobs in the tree rooted at this analysis
subtotalPrice
number Current cost (in currency
) of the job tree rooted at this analysis
priceComputedAt
timestamp Time at which subtotalPrice
was computed
subtotalEgressInfo
mapping Information about the aggregated egress amount in bytes associated with all jobs in the tree rooted at this analysis
subtotalRegionLocalEgress
int Amount in bytes of data transfer between IP in the same cloud region.
subtotalInternetEgress
int Amount in bytes of data transfer to IP outside of the cloud provider.
subtotalInterRegionEgress
int Amount in bytes of data transfer to IP in other regions of the cloud provider.
egressComputedAt
timestamp Time at which subtotalEgress
was computed
The following fields will be returned if the corresponding field in the fields
input is set to true
:
runStageSystemRequirements
mapping or null Similar torunSystemRequirements
but forstageSystemRequirements.
runSystemRequirementsByExecutable
mapping or null Similar to runSystemRequirements
but for systemRequirementsByExecutable
.
Errors
ResourceNotFound
The specified object does not exist
PermissionDenied
User does not have VIEW access to the analysis's project context
InvalidInput
Input is not a hash
fields
(if present) is not a hash or has a non-boolean key (other than stages
)
fields
has the key stages
and is not a boolean nor a hash
/analysis-xxxx/addTags
Specification
Adds the specified tags to the specified analysis. If any of the tags are already present, no action is taken for those tags.
Inputs
tags
array of strings Tags to be added
Outputs
id
string ID of the manipulated analysis
Errors
InvalidInput
The input is not a hash
The key tags
is missing, or its value is not an array, or the array contains at least one invalid (not a string of nonzero length) tag
ResourceNotFound
The specified analysis does not exist
PermissionDenied
CONTRIBUTE access is required for the analysis's project context; otherwise, the request can also be made by jobs sharing the same workspace as the parent job of the specified analysis
/analysis-xxxx/removeTags
Specification
Removes the specified tags from the specified analysis. Ensures that the specified tags are not part of the analysis -- if any of the tags are already missing, no action is taken for those tags.
Inputs
tags
array of strings Tags to be removed
Outputs
id
string ID of the manipulated analysis
Errors
InvalidInput
The input is not a hash
The key tags
is missing, or its value is not an array, or the array contains at least one invalid (not a string of nonzero length) tag
ResourceNotFound
The specified analysis does not exist
PermissionDenied
CONTRIBUTE access is required for the analysis's project context; otherwise, the request can also be made by jobs sharing the same workspace as the parent job of the specified analysis
/analysis-xxxx/setProperties
Specification
Sets properties on the specified analysis. To remove a property altogether, its value needs to be set to the JSON null (instead of a string). This call updates the properties of the analysis by merging any old (previously existing) ones with what is provided in the input, the newer ones taking precedence when the same key appears in the old.
Best practices: to completely "reset" properties (i.e. remove all existing key/value pairs and replace them with some new), issue a describe call to get the names of all properties, then issue a setProperties request to set the values of those properties to null.
Inputs
properties
mapping Properties to modify
key Name of property to modify
value string or null Either a new string value for the property, or null to unset the property
Outputs
id
string ID of the manipulated analysis
Errors
InvalidInput
There exists at least one value in properties
which is neither a string nor the JSON null
For each property key-value pair, the size, encoded in UTF-8, of the property key may not exceed 100 bytes and the property value may not exceed 700 bytes
ResourceNotFound
The specified analysis does not exist
PermissionDenied
CONTRIBUTE access is required for the analysis's project context; otherwise, the request can also be made by jobs sharing the same workspace as the parent job of the specified analysis
/analysis-xxxx/terminate
Specification
Terminates an analysis and all of the stages' origin jobs and/or analyses. This call is only valid from outside the platform.
Analyses can only be terminated by the user who launched the analysis and has at least CONTRIBUTE access or by any user with ADMINISTER access to the project context.
Inputs
None
Outputs
id
string ID of the terminated analysis (i.e., the string “analysis-xxxx”)
Errors
ResourceNotFound
The specified object does not exist
PermissionDenied
ADMINISTER access required to the project context of the job or else the user must match the “launchedBy” entry of the analysis object
InvalidState
The analysis is not in a state from which it can be terminated, e.g. it is in a terminal state
/analysis-xxxx/update
Specification
Updates an analysis and all of its stages' jobs and/or analyses. This call is only valid from outside the platform. Currently, only rank may be updated, and only root analyses may be updated.
A valid rank field must be provided. The organization associated with this analysis must have the license feature executionRankEnabled
active in order to update rank. In addition, the user must be the original launcher of the analysis or is an administrator of the organization.
When supplying rank the job/analysis being updated must be a rootExecution
, and the job/analysis being updated must be in a state that is capable of creating more jobs. Rank cannot be supplied for terminal states like "terminated", "done", "failed", "debug_hold".
Inputs
rank
integer The rank to set the analysis and all of its children executions to.
Outputs
id
string ID of the updated analysis (i.e., the string "analysis-xxxx")
Errors
InvalidInput
Input is not a hash
Expected input to have property "rank"
Expected key "rank" of input to be an integer
Expected key "rank" of input to be in range [-1024, 1023]
Not a root execution
PermissionDenied
billTo
does not have license feature executionRankEnabled
Not permitted to change rank
ResourceNotFound
The specified object does not exist
Most of the above options can also be set when the stage is created and can always be modified afterwards via the method.
Stages can be reordered or removed using the and API methods. As mentioned previously, both the stage ID and the workflow's edit version will need to be provided in order to modify them.
Replacing the executable of a stage in-place (keeping all other metadata associated with the stage such as its name, output folder, bound inputs, etc.), can only be done using the API method. This method will test whether the replacement candidate has input and output specifications which are fully compatible with the previous executable if it is still accessible. If it is not completely compatible, it can still be updated by setting the force
flag to true, in which case the workflow will also be updated to remove any outdated links between stages and other such outdated metadata.
A stage ID uniquely identifies the stage within a workflow and allows inputs and outputs of different stages to be linked to each other. When adding a stage (either in or ) you must supply a unique ID to identify each stage. As an exception, in it is not mandatory to supply an ID; if you do not do so, an arbitrary unique ID will be generated on your behalf.
The workflow can have a default output folder which is set by its outputFolder
field (either at workflow creation time or through the method). This value can be overridden at runtime using the folder
field. If no value for the output folder can be found in the API call nor in the workflow, then the system default of "/" is used.
Each stage can also specify its default output folder. This can be defined relative to the worflow's output folder, or as an absolute path. This field can be set in the method and further updated using the method.
If the value set for the stage's folder
field starts with the character "/", then this is interpreted as an absolute path that will be used for the stage's outputs, regardless of what is provided as folder
in the method.
If, however, the value set for the field does not start with the character "/", then it is interpreted as a path relative to the field folder
provided to method.
It is possible to define an explicit input to the workflow by specifying inputs
for the method, for example:
The outputs of some or all of the stages can be defined to be the output of the workflow. In order to do that, the field outputs
needs to be passed to , which defines references to stages' outputs in outputSource
. For example, if we'd like the output of the workflow to be the output of "outputFieldName" of the stage stage-xxxx
, but the output of other stages are of no interest to us, we can define it in the following way:
When adding an executable as a stage or modifying it using the API method, you can choose to specify values for some or all of the stage inputs. These bound inputs can be overridden when the workflow is actually run. The syntax for providing bound input is the same as when providing an input hash to run the executable directly. For example, you can set the input for a stage with the hash:
See the section on for more information.
The API method can also be used to modify how an input or output to a stage can be represented as an input or output of the workflow. For example, a particular input parameter can be hidden so that it does not appear in the inputSpec
field when describing the workflow. Or, it can be given a name (unique in the workflow) so that its stage does not have to be specified when providing input to the workflow. Its label or help can also be changed to document how it may interact with other stages in the workflow.
Each stage can have an executionPolicy
field to request the value to be passed on when the stage is run (see the executionPolicy
field in the of apps and applets for the accepted options).
These stored execution policies can also change the failure propagation behavior. By default, if a stage fails, the entire analysis will enter the "partially_failed" state, and other stages will be allowed to finish successfully if they are not dependent on the failed stage(s). This behavior can be modified to propagate failure to all other stages by setting the onNonRestartableFailure
flag in the executionPolicy
field for an individual stage to have value "failAllStages". These stage-specific options can also be overridden at runtime by providing a single value to be used by all stages in the call.
Each stage of the workflow can have a systemRequirements
field to request certain instance types by default when the workflow is run. This field uses the same syntax as used in the for applets and apps. This value can be set when the stage is added or modified afterwards with the API method.
These stored defaults can be further overridden (in part or in full) at runtime by providing some combination of systemRequirementsByExecutable
, systemRequirements
and stageSystemRequirements
fields in . In particular, the value for a particular entry point in a stage's stored value for systemRequirements
will still hold unless it is overridden either explicitly (via a new value for the same entry point name) or implicitly (via a value for the "*" entry point). Refer to the information in for more details.
A license is required to use the Smart Reuse feature. for more information.
When running a workflow, if the Smart Reuse feature has been enabled, the system will attempt to reuse previously computed results by looking up analyses that have been created for the workflow. To find out which stages have cached results on hand without running the workflow, you can call the method or with method with getRerunInfo
set to true. To turn off this automatic behavior, you can request that certain stages be forcibly rerun using rerunStages
in the method.
for more on this feature.
Thus if the first stage has ID "stage-xxxx" and would run an executable which takes in an input named "reads", then to provide the input for this parameter, you would use the key "stage-xxxx.reads" in the input hash. These names can be renamed via the API call using the stages.stage-xxxx.inputSpecMods
field.
outputFolder
string (optional) The default output folder for the workflow; see the section above for more details on how it interacts with stages' output folders
details
mapping or array (optional, default { }) JSON object or array that is to be associated with the object; see the section for details on valid input
inputs
array of mappings (optional) An input specification of the workflow as described in the section
outputs
array of mappings (optional) An output specification of the workflow as described in the section with an additional field specifying outputSource
; see the section for details
id
string ID that uniquely identifies the stage. See the section on for more information
folder
string (optional, default is null) The output folder into which outputs should be cloned for the stage; see the section above for more details
input
mapping (optional) The inputs to this stage to be bound. See the section on for more information.
executionPolicy
mapping (optional) A collection of options that govern automatic job restart upon certain types of failures; this can only be set at the user-level API call (jobs cannot override this for their subjobs). Contents of this field will override any of the corresponding keys in the executionPolicy
mapping found in the executable's (if present). Includes the following optional key/values:
systemRequirements
mapping (optional) Request specific resources for the stage's executable; see the section for more details
nonce
string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single workflow is created. For more information, see .
treeTurnaroundTimeThreshold
integer (optional, default: The treeTurnaroundTimeThreshold
field of the initializeFrom
workflow if the billTo
of the project has jobNotifications
enabled and initializeFrom
is not N/A, otherwise N/A, with N/A meaning not available.) The turnaround time threshold (in seconds) for trees (specifically, root executions) that run this executable. See for more information about turnaround time and managing job notifications.
A license is required to use the jobNotifications
feature. Contact to enable jobNotifications
.
id
string (optional) ID that uniquely identifies the stage. If not provided, a system-generated stage ID will be set. See the section on for more information
folder
string (optional, default is null) The output folder into which outputs should be cloned for the stage; see the section above for more details
input
mapping (optional) A subset of the inputs to this stage to be bound. See the section on for more information. key Input field name value Input field value
executionPolicy
mapping (optional) A collection of options that govern automatic job restart upon certain types of failures; this can only be set at the user-level API call (jobs cannot override this for their subjobs). Contents of this field will override any of the corresponding keys in the executionPolicy
mapping found in the executable's (if present). Includes the following optional key/values:
systemRequirements
mapping (optional) Request specific resources for the stage's executable; see the section for more details
outputFolder
string or null (optional) The default output folder for the workflow, or null to unset; see the section above for more details on how it interacts with stages' output folders
inputs
array of mappings or null (optional) An input specification of the workflow as described in the section
outputs
array of mappings or null (optional) An output specification of the workflow as described in the section with an additional field specifying outputSource
; see the section for details
folder
string or null (optional) The output folder into which outputs should be cloned for the stage; see the section above for more details; use null to unset the folder
input
mapping (optional) A subset of the inputs to this stage to be bound or unbound (using null to unset a previously-bound input). See the section on for more information. key Input field name from this stage's executable value Input field value, or null to unset
stageRequirements
mapping (optional) Request specific resources for the stage's executable; see the section for more details; use the empty mapping { } to unset
Alternatively, you can use the method to describe a large number of data objects at once.
outputFolder
string or null The default output folder for the workflow, or null if unset; see the section above for more details on how it interacts with stages' output folders
inputSpec
array of mappings, or null The value is null if a stage's executable is inaccessible; otherwise, the value is the effective input specification for the workflow. This is generated automatically, taking into account the stages' input specifications and any modifications that have been made to them in the context of the workflow (see the field inputSpecMods
under the specification for the API method). If not otherwise modified via the API, the group name of an input field will be transformed to include a prefix using its stage ID. Hidden parameters are not included unless requested via includeHidden
. They will have a flag hidden
set to true
. Bound inputs will always show up as default
values for the respective input fields.
outputSpec
array of mappings, or null The value is null if a stage's executable is inaccessible; otherwise, the value is effective output specification for the workflow. This is generated automatically, taking into account the stages' output specifications and any modifications that have been made to them in the context of the workflow (see the field outputSpecMods
under the specification for the API method). Hidden parameters are not included unless requested via includeHidden
, and they will have a flag hidden
set to true
.
folder
string or null The output folder into which outputs should be cloned for the stage; see the section above for more details; null if not set
treeTurnaroundTimeThreshold
integer or null The turnaround time threshold (in seconds) for trees (specifically, root executions) that run this executable. See for more information about turnaround time and managing job notifications.
A license is required to use the jobNotifications
feature. Contact to enable jobNotifications
.
project
string (required if invoked by a user; optional if invoked from a job with detach: true
option; prohibited when invoked from a job with detach: false
) The ID of the project in which this workflow will be run (i.e., the project context). If invoked with the detach: true
option, then the detached analysis will run under the provided project
(if provided), otherwise project context is inherited from that of the invoking job. If invoked by a user or run as detached, all output objects are cloned into the project context; otherwise, all output objects will be cloned into the temporary workspace of the invoking job. See for more information.
A license is required to launch detached executions. for more information.
folder
string (optional) The folder into which objects output by the analysis will be placed. If the folder does not exist when the job(s) complete, it (and any parent folders necessary) will be created. See the section above for more details on how it interacts with stages' output folders. If no value is provided here and the workflow does not have outputFolder
set, then the default value is "/".
stageFolders
mapping (optional) Override any stored options for the workflow stages' folder
fields. See the section for more details
executionPolicy
mapping (optional) A collection of options that govern automatic job restart upon certain types of failures; this can only be set at the user-level API call (jobs cannot override this for their subjobs). Contents of this field will override any of the corresponding keys in the executionPolicy
mapping found in individual stages and their executables' (if present). Includes the following optional key/values:
systemRequirements
mapping (optional) Request specific resources for all stages not explicitly specified in stageSystemRequirements
; values will be merged with stages' stored values as described in the section. See the section for more details
stageSystemRequirements
mapping (optional) Request specific resources by stage; values will be merged with stages' stored values as described in the section
systemRequirementsByExecutable
mapping (optional) Request system requirements for all jobs in the resulting execution tree, configurable by executable and by entry point, described in more detail in the section.
timeoutPolicyByExecutable
mapping (optional) The timeout policies for jobs in the resulting job execution tree, configurable by executable and the entry point within that executable. See thetimeoutPolicyByExecutable
field in for more details.
nonce
string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single analysis is created. For more information, see .
A license is required to be able to launch detached executions. for more information.
A license is required to use the Job Ranking feature. for more information.
A license is required to usepreserveJobOutputs
. for more information.
detailedJobMetrics
boolean Requests detailed metrics collection for jobs if set to true. The default value for this flag is project billTo
's detailedJobMetricsCollectDefault
policy setting or false if org default is not set. This flag can be specified for root executions and will apply to all jobs in the root execution. The list of detailed metrics collected every 60 seconds and viewable for 15 days from the start of a job is .
A license is required to use detailedJobMetrics
. Contactfor more information.
For InvalidInput errors that result from a mismatch of an applet or app’s input specification, an additional field is provided in the error JSON of the form (see documentation for for more details.
Perform a dry run of the API method.
Same as would be provided to
Same as the output if the resulting analysis had been described (see )
Same as would be thrown if had been called with the same input
stages
array of mappings List of metadata for each stage; see description in for more details on what may be returned in each element of the list
treeTurnaroundTime
integer The turnaround time (in seconds) of this root execution, which is the time between its creation time and its terminal-state time (or the current time if it is not in a terminal state. Terminal states for an execution include done, terminated, and failed. See for information on them). If this root execution can be retried, the turnaround time begins at the creation time of the root execution's first try, so it includes the turnaround times of all tries.
A license is required to use the jobNotifications
feature. Contact to enable jobNotifications
.
runSystemRequirements
mapping or null A mapping with thesystemRequirements
values that were passed explicitly to or when this analysis was created, or null
if the systemRequirements
input was not supplied the API call that created this analysis.
mergedSystemRequirementsByExecutable
mapping or null A mapping with values of systemRequirementsByExecutable
supplied to all the ancestors of this analysis and the value supplied to create this analysis, merged as described in the section. If neither the ancestors of this analysis nor this analysis itself were created with the systemRequirementsByExecutable
input, mergedSystemRequirementsByExecutable
value of null
is returned.
A license is required to use the Job Ranking feature. for more information.