Introduction to Building Workflows

Creating a workflow is easiest via the web interface, but you can use the DNAnexus SDK, dx-toolkit, if you want to automate workflow creation or lock down your workflow. In this tutorial we show you how to do that step by step from your local workstation.

For information on building Nextflow workflows, see Running Nextflow Pipelines.

Simple Workflows

A workflow can be created on the DNAnexus Platform from a dxworkflow.json file.

In this tutorial, we build a workflow named "BWA MEM + FreeBayes Exome Workflow". The stages field of our JSON file holds a list of executables for the workflow. We add two stages to our workflow: the first one runs the app BWA-MEM FASTQ Read Mapper and the second one runs FreeBayes Variant Caller. We also specify a name and an output folder where our results are saved. Our dxworkflow.json looks as follows:

The dxworkflow.json file in this example contains two separate fields: sorted_bams and sorted_bam. The sorted_bams input field for the FreeBayes app is bound to the sorted_bam field of the BWA step.

{
  "name": "BWA MEM + FreeBayes Exome Workflow ",
  "outputFolder": "/results",
  "stages": [
    {
      "id": "align_reads",
      "executable": "app-bwa_mem_fastq_read_mapper",
      "input": {
        "genomeindex_targz": {
          "$dnanexus_link": {
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY4942J35xX095VZyQBk0v"
          }
        }
      }
    },
    {
      "id": "call_variants",
      "executable": "app-freebayes",
      "input": {
        "sorted_bams": [{
          "$dnanexus_link": {
            "stage": "align_reads",
            "outputField": "sorted_bam"
          }
        }],
        "genome_fastagz": {
          "$dnanexus_link":{
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
          }
        }
      }
    }
  ]
}

Each stage in the stages list should have an id, which is a free-form string unique in a given workflow, and an executable field, which holds either the ID/name of an app, or an ID of an applet that we want to run in that stage.

We can add an input field for a stage if we want to bind the input of that stage with an output/input of a different stage. For example, the file array input sorted_bams of our second stage, call_variants, receives values from the output field sorted_bam of the first stage, align_reads:

{
  "input": {
    "sorted_bams": [{
      "$dnanexus_link": {
        "stage": "align_reads",
        "outputField": "sorted_bam"
      }
   }]
  }
}

These input and output field names are not created by the user, but are defined by the apps or applets they belong to. For apps, these field names can be found in each app's documentation, which is available in the online interface using the Tools Library.

To view the names of the input and output fields of an executable (app or applet) defined for a stage, run the dx describe command in the command line.

We also use the input section of a stage to set default values for a field. We select the file hs37d5.bwa-index.tar.gz (file-B6ZY4942J35xX095VZyQBk0v), which is publicly available in the reference project "Apps Data: AWS US (East)" (project-BQpp3Y804Y0xbyG4GJPQ01xv) on the DNAnexus Platform, to be the default reference file for the alignment step, align_reads.

"input": {
  "genomeindex_targz": {
    "$dnanexus_link": {
      "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
      "id": "file-B6ZY4942J35xX095VZyQBk0v"
    }
  }
}

Creating a Workflow on the DNAnexus Platform

We can create a workflow object in the DNAnexus Platform by following these steps:

Create a directory named "BWA MEM + FreeBayes Exome Workflow" in your local workstation. The directory name does not have to be identical to the name of the workflow, but it's a good practice to keep them the same.
Place your dxworkflow.json file in the newly created directory.
To create the workflow on the DNAnexus Platform, navigate to your directory and then enter the following commands:

$ ls "BWA MEM + FreeBayes Exome Workflow"
dxworkflow.json
$ dx build "BWA MEM + FreeBayes Exome Workflow"

You should see the ID of your resulting workflow in the command line. You can also view this workflow by logging in to your DNAnexus account on the Platform and viewing the workflow from your project's Manage page.

When we want to run a workflow, we can pass or override values to any stage inputs:

dx run -i align_reads.reads_fastqgzs=myreads.fastq.gz \
  -i align_reads.genomeindex_targz=file-xxxx \
  "BWA MEM + FreeBayes Exome Workflow"

Locked Workflows

Reasons to Lock a Workflow

Sometimes, you may want to prevent users from changing certain stage inputs in a workflow. For example, you might want to ensure that only a specific reference genome is used, and restrict users from modifying the reference genome input.

To achieve that we can add explicit fields called inputs and outputs to the workflow during creation, with links to inputs and outputs of specific stages. When the workflow is run, the user can pass values only to the fields defined in inputs, and all the parameters that are not visible in this workflow-level I/O interface are unchangeable/non-overridable.

Creating locked workflows is also useful when we want to simplify the workflow execution and make it clear which inputs users are expected to provide.

This feature also makes the execution of WDL workflows on the Platform more seamless since these also explicitly specify workflow inputs and outputs.

Building a Locked-Down Workflow

For our example, we create a locked down version of the workflow above and name it "BWA MEM + FreeBayes Exome Workflow (locked)". Our workflow has all inputs locked except for one in the stage align_reads, reads_fastqgzs. When locking workflows we always define those inputs that are not locked, by listing them in the workflow-level inputs field. All the other inputs are automatically locked and users cannot override their values when running this workflow.

Inputs

To create a locked workflow we first need to add a workflow-level input specification in the inputs field, which may look like this:

{
  "inputs": [
    {
      "name": "reads",
      "help": "An array of files, in gzipped FASTQ format.",
      "class": "array:file",
      "patterns": [ "*.fq.gz", "*.fastq.gz" ]
    }
  ]
}

In this case the workflow has only one input, named reads.

Stages

Next, we should define which stage or stages consume that input by adding a link from those stages to the workflow input. We can do this by using the field workflowInputField, as in the example below. If a file is supplied to reads when the workflow is run, it is directed to reads_fastqgzs of the stage align_reads.

{
  "stages": [
    {
      "id": "align_reads",
      "name": "BWA MEM",
      "executable": "app-bwa_mem_fastq_read_mapper",
      "input": {
        "reads_fastqgzs": {
          "$dnanexus_link": {
            "workflowInputField": "reads"
          }
        },
        "genomeindex_targz": {
          "$dnanexus_link": {
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY4942J35xX095VZyQBk0v"
          }
        }
      }
    },
    {
      "id": "call_variants",
      "name": "FreeBayes",
      "executable": "app-freebayes",
      "folder": "call_variants_output",
      "input": {
        "sorted_bams": [{
          "$dnanexus_link": {
            "stage": "align_reads",
            "outputField": "sorted_bam"
          }
        }],
        "genome_fastagz": {
          "$dnanexus_link":{
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
          }
        }
      }
    }
  ]
}

Notice that the input fields genomeindex_targz and genome_fastagz are not put into the workflow-level input field, indicating that these fields are locked. Since the user cannot input values for locked fields, we have to set the value for these fields using each individual stage's input field as above (not the workflow-level inputs), and the workflow only runs with the values file-B6ZY4942J35xX095VZyQBk0v and file-B6ZY7VG2J35Vfvpkj8y0KZ01 respectively.

Required Inputs

Any required stage inputs in a locked workflow must be specified in the dxworkflow.json. In the example, our stages have the following required inputs:

align_reads stage has the inputs reads_fastqgzs and genomeindex_targz.
call_variants stage has the inputs sorted_bams and genome_fastagz.

The reads_fastqgzs input is created as a workflow-level input in inputs. This means it's not locked and the user sets the input, whereas the remaining inputs are locked. The values used in the locked input fields must be set by the creator of the workflow, and for our example, they have been set according to the code snippet seen above.

If the workflow-level inputs specification is null or not specified at all, the workflow can accept inputs provided directly to the workflow stages by the user.

Multiple stages can also link to the same workflow-level input.

Outputs

Optionally, we can also specify workflow-level outputs:

{
  "outputs": [
    {
      "name": "variants",
      "class": "file",
      "outputSource": {
        "$dnanexus_link": {
          "stage": "call_variants",
          "outputField": "variants_vcfgz"
        }
      }
    }
  ]
}

The field outputSource allows us to configure which stage-level outputs are the outputs of the workflow. This, together with inputs, is especially useful when we want to set a workflow as an executable within another workflow.

Full JSON Description of a Locked-Down Workflow

Our example dxworkflow.json workflow description looks as follows:

{
  "name": "BWA MEM + FreeBayes Exome Workflow (locked)",
  "outputFolder": "/results",
  "inputs": [
    {
      "name": "reads",
      "label": "Reads",
      "help": "An array of files, in gzipped FASTQ format.",
      "class": "array:file",
      "patterns": [
        "*.fq.gz",
        "*.fastq.gz"
      ]
    }
  ],
  "stages": [
    {
      "id": "align_reads",
      "name": "BWA MEM",
      "executable": "app-bwa_mem_fastq_read_mapper",
      "input": {
        "reads_fastqgzs": {
          "$dnanexus_link": {
            "workflowInputField": "reads"
          }
        },
        "genomeindex_targz": {
          "$dnanexus_link": {
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY4942J35xX095VZyQBk0v"
          }
        }
      }
    },
    {
      "id": "call_variants",
      "name": "FreeBayes",
      "executable": "app-freebayes",
      "folder": "call_variants_output",
      "input": {
        "sorted_bams": [{
          "$dnanexus_link": {
            "stage": "align_reads",
            "outputField": "sorted_bam"
          }
        }],
        "genome_fastagz": {
          "$dnanexus_link":{
            "project": "project-BQpp3Y804Y0xbyG4GJPQ01xv",
            "id": "file-B6ZY7VG2J35Vfvpkj8y0KZ01"
          }
        }
      }
    }
  ],
  "outputs": [
    {
      "name": "variants",
      "class": "file",
      "outputSource": {
        "$dnanexus_link": {
          "stage": "call_variants",
          "outputField": "variants_vcfgz"
        }
      }
    }
  ]
}

We can then build the workflow by running this command on the directory "BWA MEM + FreeBayes Exome Workflow (locked)" (which contains the dxworkflow.json):

dx build "BWA MEM + FreeBayes Exome Workflow (locked)"

Running a Locked-Down Workflow via the CLI

To run the workflow, we should pass a FASTQ input file to the workflow-level reads input field:

dx run "BWA MEM + FreeBayes Exome Workflow (locked)" \
  -i reads="Exome Analysis Demo":/Input/SRR504516_1.fastq.gz

Providing the input file directly to the stage, for example, -ialign_reads.reads_fastqgzs=my_input_file.fastq.gz, is impossible for locked workflows.

To find out how to run the workflow and what inputs it accepts, we can use this command:

dx run "BWA MEM + FreeBayes Exome Workflow (locked)" --help

Running a Locked-Down Workflow via the UI

In the online interface, the experience of running a locked workflow resembles an app, with inputs on the left side and outputs on the right one.

A locked workflow cannot be edited or created in the UI. We can build it in the CLI using dx get and then dx build commands.

Locking Down an Existing Workflow

To lock down an existing workflow, we get the workflow from the platform by running dx get "BWA MEM + FreeBayes Exome Workflow", add inputs to the downloaded dxworkflow.json, set workflowInputField references from stages to these inputs as explained above, and run dx build again.

Last updated 8 days ago

Was this helpful?