Global Workflows

Search…
Global Workflows
Global workflows are immutable workflows that can be versioned and published to make them available to other users. They reside in a global namespace, shared with apps.
They are implemented as wrappers of regular, project-based workflow objects so any workflow can be converted to a global workflow with a unique name and version.
Conceptually, and on the API level, global workflows are similar to apps. Both are executables that:
are global objects that are not stored in any particuar project but can be accessed by users from the DNAnexus tools library
have a unique name in the global namespace (this namespace includes both apps and global workflows)
can have multiple versions. Together with the name, versions allow users to uniquely identify the executable and access its provenance through the immutable version history
can be accessed by users or orgs listed in the authorized users list and the developers list
can be updated only by users or orgs listed in the developers list
can be labelled with a category (e.g. Structural Variation, RNA-seq, etc.) and searched by that category.
A resource container is associated with each version of a global workflow, a separate one is used for each of the enabled regions. The resource container contains the underlying (wrapped) workflow and all the dependencies necessary to run the global workflow.
When a global workflow is launched, an analysis object is created, like for any workflow on the DNAnexus platform.
Name and version
The name, together with a version, uniquely identifies a global workflow. Once someone claims a name (for an app or another global workflow), it cannot be changed even if the global workflow has been deleted.
When a developer creates a global workflow with a name and version, the DNAnexus platform will generate a unique ID of the form "globalworkflow-xxxx". The workflow can be uniquely referred to by that ID or by the alias "globalworkflow-<name>/<version>>", e.g. "globalworkflow-micromap/0.0.1".
One, and only one, version can be labelled with an additional "default" alias. It is assigned to the first version automatically and can be shifted to any other version. It allows the workflow to be referred to by its name only, e.g. "globalworkflow-micromap", then the version associated with the "default" flag is invoked.
Before a particular version of the workflow is published, it is not visible to other users on the platform except to the developers of the same global workflow.
Publishing
Once the developer calls the /globalworkflow-xxxx/publish route, the version of the workflow is made available to the users that are specified as the workflow's authorized users. Before publishing only the developers can access it.
The title, summary, and documentation can still be updated after the workflow is published but all the other metadata and functionality will remain immutable. It is not possible to "unpublish" a global workflow version.
Developers
Similar to app developers, global workflow developers is a set of users or orgs that can update any version of the workflow, create new versions of the workflow with a particular name, and release (publish) workflow versions to authorized users. They can also update the list of developers and authorized users of the workflow. Developers have access to the workflow versions that are unpublished (under development).
When a user first creates a global workflow with a particular name and version, that user is initially the sole developer.
One difference between apps and workflows is that any developer of a global workflow can update, publish, or delete it, not only the creator of the version.
As in the case of apps, the list of developers can be updated at any time and the list is applied to all (past and future) versions of the workflow.
Authorized users
A list of users or organizations that can view and run the published versions of the global workflow. If the workflow is unpublished, only developers can access it.
In order to run and view the workflow, the users must also be the authorized users of all the apps of the workflow and the apps must be published.
As in the case of apps, developers can add and remove the members of the list at any time and the user list is applied to all (past and future) versions of the workflow.
Categories
As with apps, developers can add categories to a global workflows. See App Categories section for details on app categories that also apply to global workflows.
Updating workflow versions
Unlike apps, global workflows cannot be updated in place (i.e. without changing the version and hash ID) even when they are unpublished. A version string must be changed when the developer wants to make changes to the workflow. The only exception is the metdata fields unrelated to the function of the workflow, such as title, summary, and documentation. They can be changed at any time for both published and unpublished versions of the workflow.
Deleting workflow versions
Global workflows can be marked as deleted using the /globalworkflow-xxxx[/yyyy]/delete route. After a global workflow version is deleted, it can still be described (the response of which will indicate that it has been deleted), but it can no longer be run. In order to help maintain provenance, even if all the versions of the workflow are deleted the name of the workflow remains claimed and cannot be re-used.
Workflow resource container
Just like apps, each global workflow has access to its resource container, which is used to store objects needed by the workflow. One resources container is maintained in each region in which the workflow may be run. These containers are created at global workflow creation time and are persistent as long as the workflow object exists.
When running an analysis in a certain region, the resource container of the corresponding workflow in that region is read-only to the jobs of the analysis, and it is not accessible to the user executing the workflow.
The resource containers belong to the developer. For billing purposes, data in the container is the responsibility of the user or organization specified as billTo for the global workflow.
Multi-region global workflows
A global workflow can be enabled in multiple regions. In each of the enabled regions there exists a container storing an underlying workflow that represents a region-specific executable of the global workflow. The set of enabled regions may be specified by the developer at the time the version is created, and the set is fixed thereafter for this version.
When a global workflow is executed, the region of the project context decides which region the workflow will be run in.
If the workflow contains any apps, all the apps must be enabled in the regions in which the workflow will be executed.
Global Workflow API Method Specifications
API method: /globalworkflow/new
Specification
Creates a global workflow.
A read-only container for the global workflow's resources is created and initialized as specified by the resources option. In addition, the workflow object defined under regionalOptions and the applets (together with their assets) that are defined in the workflow stages are copied into it. Developers of the global workflow have VIEW access to the resources container. After the workflow is published, the authorized users will also have VIEW access to this container.
If the workflow is being created for the first time, then the billTo will default to the billTo of the requesting user (this is the same default billing account for new projects created by the user) unless otherwise specified. This same billing account must be used for all subsequent versions created under the same global workflow name.
The fields name, version, regionalOptions (pointing to least one workflow ID in one region) are required. The set of enabled regions of a particular version will be the keys of the regionalOptions mapping. The set of enabled regions is fixed once the version is created.
Inputs
regionalOptions mapping Configurations for all regions (e.g., "aws:us-east-1", "azure:westus") in which this global workflow will be enabled
key Name of a region in which this workflow may be run
value mapping Configuration of this workflow in the corresponding region, with the following key-value pairs:
workflow string ID of the underlying workflow of this global workflow in the corresponding region. It is a regular workflow, stored in any project as a data-object. The I/O specifications of all specified underlying workflow must be equal
name string Name for the global workflow, e.g. "micromap".  Must match /^[a-zA-Z0-9._-]+$/ and cannot start with the prefix "app-" or "globalworkflow-". See Name and version.
version string A nonempty freeform string representing the global workflow’s version; must be unique for the given global name (string provided must match /^[a-zA-Z0-9._+-]+$/). See Name and version.
title string (optional) Title for the global workflow, e.g. "Micro Map". If no title (or an empty string) is found, then name is used as the title
summary string (optional) A one-line description for the global workflow
description string (optional) A longer description of the workflow, formatted in Markdown, to be displayed on the workflow's detail page)
developerNotes string (optional) More advanced documentation for the global workflow
billTo string (optional, default is billTo of the requesting user issuing the API call) ID of the entity to which all resource/storage costs associated with this global workflow will be billed; must be the ID of the requesting user or an org of which the requesting user is a member with allowBillableActivities permission. If this is a subsequent version of an existing global workflow, then billTo must be the same as billTo of the existing global workflow
nonce string (optional) Unique identifier (idempotency token) for this request. Ensures that even if multiple requests fail and are retried, only a single global workflow is created. For more information, see Nonces.
costLimit float (optional) The limit of the cost this global workflow should accrue before termination. This field will be ignored if global workflow is not a root execution.
Outputs
id string ID of this workflow (of the form "globalworkflow-<hash id>").
resourcesByRegion mapping Mapping of the resources containers created for this workflow.
key Region name (e.g., "aws:us-east-1", "azure:westus").
value string ID of the resources container of this global workflow in the corresponding region.
Errors
InvalidInput
If regionalOptions is specified, then it must be non-empty.
If workflow is specified for one region in regionalOptions, then it must be specified for all regions.
The workflow denoted by regionalOptions[r].workflow (where r is a region name) must reside in the region denoted by r
version is already in use by an existing version of this global workflows, or is a tag for a published version of this workflow.
A nonce was reused in a request but some of the other inputs had changed signifying a new and different request
A nonce may not exceed 128 bytes
PermissionDenied
VIEW access is required to the underlying workflow and to all required objects.
If the name provided is in use, then the user must also be listed as a developer for the name
A full scope token or one with developer access is required
billTo must be either the ID of the requesting user or the ID of an org of which the requesting user is a member with allowBillableActivities permission
The billTo of the invoking user is an org and the user is no longer a member of that org
The region of the workflow that this global workflow was created from must be one of the permittedRegions of this global workflows's billTo
SpendingLimitExceeded
billTo has reached its spending limit
API method: /globalworkflow-xxxx[/yyyy]/update
Specification
Updates the global workflow version with any fields that are specified below. Any developer of the global workflow can update it and unpublished as well as published versions can be updated.
Inputs
title string (optional) The global workflow’s title, e.g. "Micro Map"
summary string (optional) The global workflow’s summary
description string (optional) The global workflow’s description
developerNotes string (optional) Advanced documentation for the global workflow
Outputs
id string ID of the global workflow (of the form "globalworkflow-<hash id>")
Errors
InvalidInput
Invalid type values for any of the input fields
ResourceNotFound
The specified global workflow does not exist
PermissionDenied
User must be the developer global workflow
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/addTags
Specification
Adds the specified tags to this version of the global workflow. If any of the tags are already present, no action is taken for those tags. If any of the tags are already in use by different versions of this workflow, those tags are removed from the other versions and added to this version. This maintains the invariant that any tag only applies to a single version of a global workflow.
Example: you have two published versions of a global workflow, globalworkflow-foo/1.0.0 and globalworkflow-foo/2.0.0, and version 1.0.0 is the current default. You can make version 2.0.0 the default by calling /globalworkflow-foo/2.0.0/addTags with input {"tags":["default"]}
Inputs
tags array of strings List of tags to add to the specified global workflow
version
Outputs
id string ID of the workflow (of the form "globalworkflow-<hash id>")
Errors
ResourceNotFound
The specified global workflow does not exist
InvalidInput
Input is not a hash, tags is not present or is not an array of nonempty strings, a tag is equal to an existing version, whether published or not
PermissionDenied
User must be a developer of the global workflow
A full scope token or one with developer access is required
InvalidState (the global workflow object has already been deleted)
API method: /globalworkflow-xxxx[/yyyy]/removeTags
Specification
Remove the requested aliases of "globalworkflow-<name>/<tag>"
Inputs
tags array of strings List of tags to remove from all versions of the global workflow
Outputs
name string Global workflow name modified (e.g. "bwa")
Errors
ResourceNotFound
The specified global workflow name does not exist
InvalidInput
Input is not a hash, tags is not present or is not an array of nonempty strings, a tag is equal to an existing version, or the tag includes the special tag "default"
PermissionDenied
User must be a developer of the global workflow
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/publish
Specification
Releases the version of the global workflow so that it is now discoverable and runnable by authorized users on the platform and can be accessed by object ID, "globalworkflow-/", and "globalworkflow-/" if the global workflow object also has tags. Any developer of the global workflow can publish the version.
Inputs
makeDefault boolean (optional, default false) Whether this global workflow should take on the "default" tag, removing it from whichever workflow version was previously marked as "default"
Outputs
id string ID of the workflow (of the form "globalworkflow-<hash id>")
Errors
PermissionDenied
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/describe
Specification
Describe the global workflow.
Inputs
fields mapping (optional) Restrict the output of this method to have only the provided keys in this field
key Output field; see the "Outputs" section below for valid values here
value boolean or mapping A boolean value indicates whether to include the output field.
Outputs
id string ID of the global workflow (of the form "globalworkflow-<hash id>")
class string The value "globalworkflow"
billTo string ID of the user/organization that owns this global workflow's name (and is billed for the storage incurred by the workflow versions)
name string Name of the workflow
title string Freeform name
version string Version of the workflow
aliases array of strings List that includes the version and all tags of this workflow by which it can be addressed using "globalworkflow-/"
summary string A one-line description of the workflow
description string A long description of the workflow, formatted in Markdown, to be displayed on the workflow's detail page
categories array of strings List of categories the workflow belongs to
dxapi string The version of the API used
createdBy string ID of the user who created the workflow
created timestamp When the workflow was created
modified timestamp When this workflow was last modified
deleted boolean Whether the workflow has been deleted
isDeveloperFor boolean Whether the requesting user has developer permissions for the workflow AND the token provided has full scope or has developer access
authorizedUsers array of strings List of user and org IDs representing who can describe and run this workflow; if the string "PUBLIC" is present, then all users can access this workflow
regionalOptions mapping Mapping that contains the regional configurations for this workflow
key Region name (e.g., "aws:us-east-1", "azure:westus"). Note that the existence of a region here indicates that this workflow may be run in this region, and that absence of a region indicates otherwise.
value mapping Mapping of the configuration of this workflow in the corresponding region, with the following key-value pairs:
workflow string ID of the underlying workflow of this global workflow in the corresponding region.
resources string ID of the resources container of this global workflow in the corresponding region. It contains the underlying workflow and any other dependencies (files, applets).
If the workflow is published:
published timestamp Time at which this workflow was published
Errors
InvalidInput
Input is not a hash, or fields if present, is not a hash or has a non-boolean key
ResourceNotFound
The specified workflow does not exist
PermissionDenied
The requesting user must be represented in the access list or must be one of the workflow's developers; if the latter, a full scope token or one with developer access is required
If the workflow is unpublished, the requesting user must be one of the workflow's developers, and a full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/run
Specification
Run the specified global workflow.
Inputs
name name (optional, default is the global workflow's title if set and otherwise its name) Name for the resulting analysis
input mapping Input that the global workflow is launched with
key Input field name (as it appears in the input specification of any of the underlying workflows)
value Input field value
project string (required if invoked by a user; optional if invoked from a job with detach: true option; prohibited when invoked from a job with detach: false ) The ID of the project in which this global workflow will be run (i.e., the project context). If invoked with the detach: true option, then the detached analysis will run under the provided project (if provided), otherwise project context is inherited from that of the invoking job. If invoked by a user or run as detached, all output objects are cloned into the project context; otherwise, all output objects will be cloned into the temporary workspace of the invoking job.  See The Project Context and Temporary Workspace for more information.
folder string (optional) The folder into which objects output by the analysis will be placed.  If the folder does not exist when the job(s) complete, it (and any parent folders necessary) will be created.  See the Customizing Output Folders for more details on how it interacts with stages' output folders
stageFolders mapping (optional) Override any stored options for the workflow stages' folder fields.  See the Customizing Output Folders section for more details
key Stage ID or "*" to indicate that the value should be applied to all stages not otherwise mentioned
value null or string Value to replace the stored default
details array or mapping (optional, default { }) Any conformant JSON (i.e. a JSON object or array, per RFC4627), which will be stored with the created job
nonce string (optional) Unique identifier for this request. Ensures that even if multiple requests fail and are retried, only a single analysis is created. For more information, see Nonces.
tags array of strings (optional) Tags to associate with the resulting analysis
properties mapping (optional) Properties to associate with the resulting analysis
key Property name
value string Property value
details mapping or array (optional, default { }) JSON object or array that is to be associated with the analysis
detach boolean (optional) This option has no impact when the API is invoked by a user. If invoked from a job with detach set to true, the new analysis will be detached from the creator job and will appear as a typical root execution. A failure in the detached analysis will not cause a termination of the job from which it was created and vice versa. Detached analysis will not inherit access to the workspace of its creator job. Detached analysis' access permissions will be the intersection (most restricted) of access permissions of the creator job and the permissions requested by jobs' executables in the detached analysis.  To launch the detached analysis, creator job must have CONTRIBUTE or higher access to the project in which the detached job is launched.  Additionally, the billTo of the project in which the creator job is running must be licensed to launch detached executions.
rank integer (optional) An integer between -1024 and 1023, inclusive. The rank indicates the priority in which the executions generated from this executable will be processed. The higher the rank, the more prioritized it will be. If no rank is provided, the executions default to a rank of zero. If the execution is not a root execution, it will inherit its parent's rank. If a rank is provided, all executions relating to the workflow stages will also inherit the rank.
Outputs
id string ID of the created analysis object (i.e. a string in the form “analysis-xxxx”).
stages array of strings List of job IDs that will be created for each stage, as ordered in the workflow
Errors
ResourceNotFound
The specified workflow object, any referenced workflows, apps or applets, default input files, or project context does not exist
PermissionDenied
The requesting user must be represented in the access list or must be one of the global workflow's developers; if the latter, a full scope token or one with developer access is required
If the workflow is unpublished, the requesting user must be one of the global workflow's developers, and either (1) a full scope token or one with developer access is required or (2) the token belongs to a job that is already running the same global workflow
CONTRIBUTE access to the project context is required unless called by a job
The requesting user must be represented in the access list of all apps or must be the apps' developer; if the latter, a full scope token or one with developer access is required
When specifying allowSSH or debug options, the user must have developer access to all apps in the workflow, or the apps must have the openSource field set to true
The billTo of the job's project must be licensed to start detached executions when invoked from the job with detach: true argument.
If rank is provided and the billTo does not have license feature executionRankEnabled set to true.
InvalidInput
The workflow spec is not complete
The workflow must be enabled in the region of the project context
All data object inputs must be in the same region as the project context
All inputs that are job-based object references must refer to a job that was run in the same region as the project context.
A nonce was reused in a request but some of the other inputs had changed signifying a new and different request
A nonce may not exceed 128 bytes
InvalidState
Some input to the workflow is not in the "closed" state
The project context's region is not among the enabled regions of the workflow
API method: /globalworkflow-xxxx[/yyyy]/delete
Specification
Mark a global workflow version as deleted. A deleted version can no longer be run, modified, nor published. This state is reflected in the output of describe. The associated resources container is made inaccessible.
Inputs
None
Outputs
id string ID of the workflow (of the form "globalworkflow-<hash id>")
Errors
PermissionDenied
Requesting user must be the developer of the workflow
A full scope token or one with developer access is required
InvalidState
If the workflow has multiple versions, all other versions must be deleted before a workflow with the "default" can be deleted
API method: /globalworkflow-xxxx[/yyyy]/addDevelopers
Specification
Adds developers (users and/or orgs) to the global workflow.
If the requesting user is not a developer of the global workflow, but they are the billTo of the workflow or the billTo of the workflow is an org and the user is an ADMIN of that org, then the user may invoke this method to grant themselves, and only themselves, developer access to the workflow.
Inputs
developers array of strings List of user and/or org IDs that will be added as developers
Outputs
name string The global workflow name (e.g. "bwa")
Errors
ResourceNotFound
Specified global workflow does not exist
A specified user does not exist
InvalidInput
If the requesting user is not a developer of the workflow and is or administers the billTo of the workflow, then developers may only contain the ID of the requesting user
PermissionDenied
The requesting user must be a developer of the workflow, or they must be or administer the billTo of the workflow
If the requesting user is a developer of the workflow, then the user must have either a full-scope token or a token with developer access
If the requesting user is not a developer of the workflow but is or administers the billTo of the workflow, then the user must have a full scope token
API method: /globalworkflow-xxxx[/yyyy]/listDevelopers
Specification
List the developers of the global workflow.
Inputs
None
Outputs
developers array of strings List of user and/or org IDs that
are developers
Errors
ResourceNotFound
The specified workflow does not exist
API method: /globalworkflow-xxxx[/yyyy]/removeDevelopers
Specification
Remove a list of developers from the global workflow. Attempting to remove a user or org who is not listed as a developer will not throw an error. If the billing account of the global workflow is a user ID, then the developer access of that user may not be revoked.
Inputs
developers array of strings list of user and/or org IDs that will be removed as developers
Outputs
name string the global workflow name (e.g. "bwa")
Errors
ResourceNotFound
The specified workflow does not exist
InvalidInput
The billing account of the workflow is a user ID and developers contains that user ID
PermissionDenied
User issuing the API call must be a developer of the workflow
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/addAuthorizedUsers
Specification
Add users and/or org IDs to the access list. The access list applies to all global workflow versions.
Inputs
authorizedUsers array of strings List of user and/or org IDs to add to the access list
Outputs
name string The global workflow name (e.g. "bwa")
Errors
ResourceNotFound
A specified global workflow or user/org does not exist
InvalidInput
Input is not a hash, authorizedUsers is missing or is not an array of nonempty strings
PermissionDenied
User must be a developer of the global workflow
A full scope token or one with developer access is required
"PUBLIC" cannot be added as an authorized user. If you are interested in making an app or global workflow available to all users of the DNAnexus Platform, please contact us at [email protected]
API method: /globalworkflow-xxxx[/yyyy]/listAuthorizedUsers
Specification
List the users and/or orgs IDs on the access list for the workflow. The string "PUBLIC" represents the set of all users on DNAnexus Platform. The access list applies to all global workflow versions.
Inputs
None
Outputs
authorizedUsers array of strings List of user and/or org IDs that can describe and run the workflow; if "PUBLIC" is present, this means all users can access the workflow.
Errors
ResourceNotFound
The specified global workflow does not exist
PermissionDenied
The requesting user must be represented in the access list or must be one of the workflow's developers; if the latter, a full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/removeAuthorizedUsers
Specification
Remove a list of user and/or org IDs from the access list for the global workflow. Removing an entity that is not currently listed in the access list does not throw an error.
Inputs
authorizedUsers array of strings List of user and/or org IDs to remove from the access list
Outputs
name string The global workflow name (e.g. "bwa")
Errors
ResourceNotFound
A specified global workflow does not exist
InvalidInput
Input is not a hash, authorizedUsers is missing or is not an array of nonempty strings
PermissionDenied
User must be a developer of the global workflow
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/addCategories
Specification
Add categories for the global workflow.
Inputs
categories array of strings List of categories to add the
workflow to
Outputs
name string The global workflow name (e.g. "bwa")
Errors
ResourceNotFound
A specified global workflow does not exist
InvalidInput
Input is not a hash, categories is missing or is not an array of nonempty strings, a listed category is not allowed: "new", "recommended", or "popular"
PermissionDenied
User must be a developer of the global workflow
A full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/listCategories
Specification
List the categories that a global workflow is a member of. All authorized users can list the categories of a published workflow. Categories apply to all the versions of the global workflow.
Inputs
None
Outputs
categories array of strings List of categories the global workflow belongs to
Errors
ResourceNotFound
The specified global workflow does not exist
PermissionDenied
The requesting user must be represented in the access list or must be one of the workflow's developers; if the latter, a full scope token or one with developer access is required
If the workflow is unpublished, the requesting user must be one of the workflow's developers, and a full scope token or one with developer access is required
API method: /globalworkflow-xxxx[/yyyy]/removeCategories
Specification
Remove the categories from a global workflow.
Inputs
categories array of strings List of categories (strings) to remove from the global workflow
Outputs
name string The global workflow name (e.g. "bwa")
Errors
ResourceNotFound
A specified global workflow does not exist
InvalidInput
Input is not a hash, categories is missing or is not an array of nonempty strings
PermissionDenied
User must be a developer of the workflow
A full scope token or one with developer access is required