Versioning and Publishing Global Workflows

Learn how to create both a local, project-based workflow, and a versioned, global workflow that can be published and listed in the Platform Tools library.

Workflows and Global Workflows

Workflows created on the DNAnexus Platform with an ID in the form of workflow-xxxx are data objects stored in a project.

Because they are stored in a project, it's easy to share the workflow with other users by adding them to the project. These local workflows are great for fast iterations for development and testing. You can always delete the workflow and create a new one in its place.

A global workflow is an executable that can be versioned and published to other users. It is implemented as a wrapper around an existing project-based workflow. For users and organizations collaborating on multiple private or public projects, these local workflows may be less suitable for long-term maintenance and collaboration.

For example, an organization administrator or workflow developer may want to restrict execution of a particular workflow to specific users and organizations, or they may want to allow execution of this workflow across cloud regions and providers. Global Workflows can be used for those use cases and more.

Global Workflow Use Cases

Global workflows offer several key benefits, including:

• Version Management: Enables tracking and maintaining different versions of a workflow's source code.

• Provenance & Immutability: Provides an explicit history of changes to the workflow name and associated ID. Each version is immutable, allowing users to revert to previous states while ensuring data integrity.

• Collaboration & Sharing: Facilitates collaborative development across projects and organizations. Workflows can be shared with authorized users who have execution privileges but cannot modify the core workflow definition.

• Multi-Region Support: Allows deployment and execution of a single workflow executable across multiple geographic regions and cloud providers.

• Immutability Guarantees: Prevents modification or overriding of existing workflow versions, ensuring consistency and predictability.

• Tools Library Integration: Enables authorized users and developers to discover shared global workflows within the DNAnexus Tools Library.

Step 1: Build a Local Workflow

The easiest way to build a workflow is by using the web interface. In your project's Manage tab, click Add > New Workflow. For details, see Building and Running Workflows.

You can also use the DNAnexus SDK, dx-toolkit .

Step 2: Make the Workflow Global

Download the Original Workflow

For a local workflow you'd like to convert to a global workflow, use the dx get command to download a JSON representation (dxworkflow.json) of the local workflow:

dx get "Exome Analysis Demo:Exome Analysis Workflow"

This creates a local directory "Exome Analysis Workflow" with the dxworkflow.json file in it.

Build a Global Workflow

Adjust the downloaded dxworkflow.json before you create a new global workflow.

• Global workflows require the name and version fields.

  • The name has to be unique in the global namespace shared by apps and global workflows.

  • The name can use lowercase letters, numbers, - , . , and _ , and must not contain spaces.

• You can add fields such as title and summary to make the workflow more user-friendly.

• You can leave the stages and any other fields unchanged.

dxworkflow.json

{
  "name": "my_global_workflow",
  "title": "Exome Analysis Workflow", 
  "version":"1.0.0",
  "stages": [...]
}

It's a good practice to include documentation in the Readme.md file in the same directory as the workflow.

If the workflow uses globally available apps or applets from non-PHI projects, you can use dx build to create your global workflow. Workflows using resources from PHI-enabled projects cannot be built as global workflows due to the project's protective restrictions.

# Create your global workflow
dx build --globalworkflow "Exome Analysis Workflow"

Because Exome Analysis Workflow is the name of the directory storing the dxworkflow.json, it can be changed freely.

Describe and Run the Workflow

The dx build command returns the unique ID of the created version, such as globalworkflow-xxxx .

You can use the ID to refer to the global workflow.

dx describe globalworkflow-xxxx

Running a global workflow is the same as running any other workflow.

dx run my_global_workflow \
  -i0.reads_fastqgzs="Exome Analysis Demo:/Input/SRR504516_1.fastq.gz"

Step 3: Add Authorized Users or Orgs

You can specify a list of users with whom you want to share the workflow by using dx add users.

The users will be able to find and access the global workflow only after the workflow is published. The list of users can be updated by developers before or after it is published, and it will apply to all to the versions of the workflows, past and future.

# Share the workflow with a user AND with an organization
dx add users my_global_workflow user-bob org-partnerorg

# View who's on the workflow's access list
dx list users my_global_workflow

# Remove a user from the access list
dx remove users my_global_workflow org-partnerorg

The authorized user permissions do not propagate to apps automatically. If the workflow contains any apps, the users need access to those apps in order to run the workflow. To add users to an app, you can use the same dx add users command.

Step 4: Publish a Version to Users

Once you've tested the workflow version, you can release it to the authorized users using dx publish .

dx publish my_global_workflow/0.0.1

Publishing a workflow version has the following effects:

• The authorized users can discover the version via dx find globalworkflow .

• The authorized users can describe and run the version.

• The authorized users can download the workflow's dxworkflow.json source code with dx get (as well as the dependencies, e.g. applets), though they cannot build new versions with the same global workflow name.

Developers can add and remove users and other developers at any time. Adding users to the workflow will give the users access to all the published versions but it will have no effect on the unpublished versions. Unpublished versions are only accessible to the people listed as developers of the workflow.

The "Default" Alias

Executing dx publish command makes the published version a default one by adding a "default" alias to this version (and removing it from whichever workflow version was previously marked as "default").

The "default" alias indicates that this version will be invoked whenever the workflow name is used without specifying a version. For example, users can run the workflow:

dx run my_global_workflow   # equivalent to "dx run my_global_workflow/0.0.1"

The first created version of a global workflow gets "default" flag automatically.

Next Steps

Add Developers

You can enable multiple users to update the current version or build new versions of the global workflow.

# Add a user to the workflow's developer list
dx add developers my_global_workflow user-eve

Either specific users or whole orgs can be developers of a global workflow.

# See who's on the workflow's developer list
dx list developers my_global_workflow

# Remove a user from the developer list
dx remove developers my_global_workflow user-bob

Create a New Version

You can create a new version of the workflow by updating the dxworkflow.json and using dx build --globalworkflow. To create the new version, you need to explicitly update the version string, for example, to "0.0.2".

If you lose the source dxworkflow.json, you can download it at any time using dx get .

# Download dxworkflow.json for a specific version of a global workflow 
dx get globalworkflow-my_global_workflow/0.0.2

Any developer of the workflow can download its source code and build new versions. Authorized users that have access to published versions can only download or run it.

Delete

You can mark a global workflow version as deleted, which will make the version unrunnable. It will still be possible to describe the workflow for provenance purposes.

dx api globalworkflow-xxxx delete

Use with caution

• Deleting a global executable can break users' reproducibility requirements.

• Deleting all workflow versions will not release the workflow name. It will not be possible to reuse the name for a different global workflow by another user.

Discovering Global Workflows

Searching Available Versions

You can use the command dx find globalworkflows to browse the global workflows available to you.

# List the current default versions of published global workflows  
dx find globalworkflows

# List the complete version history of published global workflows
dx find globalworkflows --all

# List the global workflows that are not published
dx find globalworkflows --unpublished --all

Searching by Category

You can add the --category parameter to restrict the search to a specific category.

Common categories are available as tab completions. For example:

dx find globalworkflows --category Variation\ Calling

To view all available categories you can search by:

dx find globalworkflows --category-help

Summary

The following table summarizes the main steps and stages required to create and publish the workflow.

Object name	ID prefix	Namespace	Access	Definition and purpose
Workflow	workflow	Project	based on project permissions	A file-like workflow object stored in a project; used for private, light-weight development
Global workflow unpublished	globalworkflow	Global	developers	A development version of a workflow that is not visible to users but is to developers; used for development and testing in multiple regions, release management, preparation for publishing to users
Global workflow published	globalworkflow	Global	users, developers	A version of a workflow that is available to authorized users
Global workflow deleted	globalworkflow	Global	users, developers	A version (previously published or unpublished) that is not runnable but can be described