Introduction to Building Workflows
Creating a workflow is easiest via the web interface, but you can use the DNAnexus SDK, dx-toolkit
, if you want to automate workflow creation or lock down your workflow. In this tutorial we will show you how to do that step by step from your local workstation.
For information on building Nextflow workflows, see Running Nextflow Pipelines.
Simple Workflows
A workflow can be created on the DNAnexus Platform from a dxworkflow.json
file.
In this tutorial, we will build a workflow named "BWA MEM + Freebayes Exome Workflow". The stages
field of our JSON file holds a list of executables for the workflow. We'll add two stages to our workflow: the first one will run the app BWA-MEM FASTQ Read Mapper and the second one will run FreeBayes Variant Caller. We'll also specify a name and an output folder where our results will be saved. Our dxworkflow.json
will look as follows:
{
"name": "BWA MEM + Freebayes Exome Workflow ",
"outputFolder": "/results",
"stages": [
{
"id": "align_reads",
"executable": "app-bwa_mem_fastq_read_mapper",
"input": {
"genomeindex_targz": {
"$dnanexus_link": {
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY4942J35xX095VZyQBk0v"
}
}
}
},
{
"id": "call_variants",
"executable": "app-freebayes",
"input": {
"sorted_bams": [{
"$dnanexus_link": {
"stage": "align_reads",
"outputField": "sorted_bam"
}
}],
"genome_fastagz": {
"$dnanexus_link":{
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
}
}
}
}
]
}
Each stage in the stages
list should have an id
, which is a free-form string unique in a given workflow, and an executable
field, which holds either the ID/name of an app, or an ID of an applet that we want to run in that stage.
We can add an input
field for a stage if we want to bind the input of that stage with an output/input of a different stage. For example, the file array input sorted_bams
of our second stage, call_variants
, will receive values from the output field sorted_bam
of the first stage, align_reads
:
{
"input": {
"sorted_bams": [{
"$dnanexus_link": {
"stage": "align_reads",
"outputField": "sorted_bam"
}
}]
}
}
We also use the input
section of a stage to set default values for a field. We select the file hs37d5.bwa-index.tar.gz
(file-B6ZY4942J35xX095VZyQBk0v
), which is publicly available in the reference project "Apps Data: AWS US (East)" (project-BQpp3Y804Y0xbyG4GJPQ01xv
) on the DNAnexus Platform, to be the default reference file for the alignment step, align_reads
.
"input": {
"genomeindex_targz": {
"$dnanexus_link": {
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY4942J35xX095VZyQBk0v"
}
}
}
Creating a Workflow on the DNAnexus Platform
We can now create a workflow object in the DNAnexus Platform by following these steps:
Create a directory named "BWA MEM + Freebayes Exome Workflow" in your local workstation. The directory name does not have to be identical to the name of the workflow, but it's a good practice to keep them the same.
Place your
dxworkflow.json
file in the newly created directory.To create the workflow on the DNAnexus Platform, navigate to your directory and then enter the following commands:
$ ls "BWA MEM + Freebayes Exome Workflow"
dxworkflow.json
$ dx build "BWA MEM + Freebayes Exome Workflow"
You should see the ID of your resulting workflow in the command line. You can also view this workflow by logging in to your DNAnexus account on the Platform and viewing the workflow from your project's Manage page.
When we want to run a workflow, we can pass or override values to any stage inputs:
dx run -i align_reads.reads_fastqgzs=myreads.fastq.gz \
-i align_reads.genomeindex_targz=file-xxxx \
"BWA MEM + Freebayes Exome Workflow"
Locked Workflows
Reasons to Lock a Workflow
Sometimes, you may want to prevent users from changing certain stage inputs in a workflow. For example, you might want to ensure that only a specific reference genome is used, and restrict users from modifying the reference genome input.
To achieve that we can add explicit fields called inputs
and outputs
to the workflow during creation, with links to inputs and outputs of specific stages. When the workflow is run, the user will be able to pass values only to the fields defined in inputs
, and all the parameters that are not visible in this workflow-level I/O interface will be unchangeable/non-overridable.
Creating locked workflows is also useful when we want to simplify the workflow execution and make it clear which inputs users are expected to provide.
This feature also makes the execution of WDL workflows on the Platform more seamless since these also explicitly specify workflow inputs and outputs.
Building a Locked-Down Workflow
For our example, we will create a locked down version of the workflow above and name it "BWA MEM + Freebayes Exome Workflow (locked)". Our workflow will have all inputs locked except for one in the stage align_reads
, reads_fastqgzs
. When locking workflows we always define those inputs that are not locked, by listing them in the workflow-level inputs
field. All the other inputs will be automatically locked and users will not be able to override their values when running this workflow.
Inputs
To create a locked workflow we first need to add a workflow-level input specification in the inputs
field, which may look like this:
{
"inputs": [
{
"name": "reads",
"help": "An array of files, in gzipped FASTQ format.",
"class": "array:file",
"patterns": [ "*.fq.gz", "*.fastq.gz" ]
}
]
}
In this case the workflow will have only one input, named reads
.
Stages
Next, we should define which stage or stages will consume that input by adding a link from those stages to the workflow input. We can do this by using the field workflowInputField
, as in the example below. If a file is supplied to reads
when the workflow is run, it will be directed to reads_fastqgzs
of the stage align_reads
.
{
"stages": [
{
"id": "align_reads",
"name": "BWA MEM",
"executable": "app-bwa_mem_fastq_read_mapper",
"input": {
"reads_fastqgzs": {
"$dnanexus_link": {
"workflowInputField": "reads"
}
},
"genomeindex_targz": {
"$dnanexus_link": {
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY4942J35xX095VZyQBk0v"
}
}
}
},
{
"id": "call_variants",
"name": "Freebayes",
"executable": "app-freebayes",
"folder": "call_variants_output",
"input": {
"sorted_bams": [{
"$dnanexus_link": {
"stage": "align_reads",
"outputField": "sorted_bam"
}
}],
"genome_fastagz": {
"$dnanexus_link":{
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
}
}
}
}
]
}
Notice that the input fields genomeindex_targz
and genome_fastagz
are not put into the workflow-level input field, indicating that these fields are locked. Since the user cannot input values for locked fields, we have to set the value for these fields using each individual stage's input
field as above (not the workflow-level inputs
), and the workflow will only be run with the values file-B6ZY4942J35xX095VZyQBk0v
and file-B6ZY7VG2J35Vfvpkj8y0KZ01
respectively.
Required Inputs
Any required stage inputs in a locked workflow must be specified in the dxworkflow.json
. In the example, our stages have the following required inputs:
align_reads
stage has the inputsreads_fastqgzs
andgenomeindex_targz
.call_variants
stage has the inputssorted_bams
andgenome_fastagz
.
The reads_fastqgzs
input is created as a workflow-level input in inputs
. This means it's not locked and the user sets the input, whereas the remaining inputs are locked. The values used in the locked input fields must be set by the creator of the workflow, and for our example, they have been set according to the code snippet seen above.
If the workflow-level inputs
specification is null or not specified at all, the workflow can accept inputs provided directly to the workflow stages by the user.
Multiple stages can also link to the same workflow-level input.
Outputs
Optionally, we can also specify workflow-level outputs
:
{
"outputs": [
{
"name": "variants",
"class": "file",
"outputSource": {
"$dnanexus_link": {
"stage": "call_variants",
"outputField": "variants_vcfgz"
}
}
}
]
}
The field outputSource
allows us to configure which stage-level outputs will be the outputs of the workflow. This, together with inputs
, is especially useful when we want to set a workflow as an executable within another workflow.
Full JSON Description of a Locked-Down Workflow
Our example dxworkflow.json
workflow description will look as follows:
{
"name": "BWA MEM + Freebayes Exome Workflow (locked)",
"outputFolder": "/results",
"inputs": [
{
"name": "reads",
"label": "Reads",
"help": "An array of files, in gzipped FASTQ format.",
"class": "array:file",
"patterns": [
"*.fq.gz",
"*.fastq.gz"
]
}
],
"stages": [
{
"id": "align_reads",
"name": "BWA MEM",
"executable": "app-bwa_mem_fastq_read_mapper",
"input": {
"reads_fastqgzs": {
"$dnanexus_link": {
"workflowInputField": "reads"
}
},
"genomeindex_targz": {
"$dnanexus_link": {
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY4942J35xX095VZyQBk0v"
}
}
}
},
{
"id": "call_variants",
"name": "Freebayes",
"executable": "app-freebayes",
"folder": "call_variants_output",
"input": {
"sorted_bams": [{
"$dnanexus_link": {
"stage": "align_reads",
"outputField": "sorted_bam"
}
}],
"genome_fastagz": {
"$dnanexus_link":{
"project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
"id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
}
}
}
}
],
"outputs": [
{
"name": "variants",
"class": "file",
"outputSource": {
"$dnanexus_link": {
"stage": "call_variants",
"outputField": "variants_vcfgz"
}
}
}
]
}
We can then build the workflow by running this command on the directory "BWA MEM + Freebayes Exome Workflow (locked)" (which contains the dxworkflow.json
):
dx build "BWA MEM + Freebayes Exome Workflow (locked)"
Running a Locked-Down Workflow via the CLI
To run the workflow, we should pass a FASTQ input file to the workflow-level reads
input field:
dx run "BWA MEM + Freebayes Exome Workflow (locked)" \
-i reads="Exome Analysis Demo":/Input/SRR504516_1.fastq.gz
Providing the input file directly to the stage, for example, -ialign_reads.reads_fastqgzs=my_input_file.fastq.gz
, is impossible for locked workflows.
To find out how to run the workflow and what inputs it accepts, we can use this command:
dx run "BWA MEM + Freebayes Exome Workflow (locked)" --help
Running a Locked-Down Workflow via the UI
In the online interface, the experience of running a locked workflow will resemble an app, with inputs on the left side and outputs on the right one.
A locked workflow currently cannot be edited or created in the UI. We can build it in the CLI using dx get
and then dx build
commands.
Locking Down an Existing Workflow
To lock down an existing workflow, we get the workflow from the platform by running dx get "BWA MEM + Freebayes Exome Workflow"
, add inputs
to the downloaded dxworkflow.json
, set workflowInputField
references from stages to these inputs
as explained above, and run dx build
again.
Last updated
Was this helpful?