Building and Running Workflows
Last updated
Was this helpful?
Last updated
Was this helpful?
Workflows allow apps to be run on the DNAnexus Platform. They also allow you to create a pipeline of apps that will hand off their outputs to other apps as they complete. You can configure a complicated workflow once and when you are done the workflow will be kept so you don't have to configure it again.
Apps in a workflow will always begin executing as soon as their inputs are satisfied and if possible they will run independently. For example, if the first four apps in a workflow each depend on the previous step, but the fifth app can run immediately, then running the workflow will cause both the first and fifth apps to start processing immediately, while the second-through-fourth apps will wait for the first app to finish. For more details, see Connecting input data below.
For information on building and running Nextflow workflows, see here.
Workflows are saved as they are edited. There is no need to explicitly save a workflow.
From the Manage tab of your selected project, you can create or edit a workflow. To create a workflow, click the Add button on the upper right side the screen. From the dropdown menu, select New Workflow.
On the workflow creation screen, click Add a Step. This will bring up the list of available apps and applets.
In a step you have added, the black box with the app name indicates which app will be run in that step. To the left of that black box are a list of inputs that the app takes in. You can assign objects to inputs by clicking on one of the input boxes. This will bring up a dialog box which gives you the option to add one or more pieces of data. This dialog box will only consider inputs that match the type required by the app. If the app requires an input to have type "Mappings", then only items in your project of type "Mappings" will be shown. This means that in some cases there may be no data available in your project which meets the proper type. In this case, you will need to either first import data of that type, or generate data of that type in an earlier step in the workflow and connect the output that matches the requirements. For information on connecting data from an earlier app in the pipeline, see the section Connecting input data.
App inputs are rendered in one of several ways:
Required inputs are orange
Optional inputs are grey
Required inputs with a default that has been propagated from another input or output are displayed in a darker grey box with the name of the linked input in blue.
When you have satisfied all of the input requirements for an app, the text below the black app box will say "Runnable" in green.
Some apps require more input than just the input boxes to the left. For example, the GATK UnifiedGenotyper app contains parameters to change the behavior of the app similar to using UnifiedGenotyper locally from the command line. To access these parameters, click on the black app box. This will bring up a box of parameters. In some cases, there will be a tab of advanced parameters.
You can connect multiple apps together by dragging the output of one app to the input of another. Once you start to drag an output or input, valid places where that item can be placed will be highlighted. If a second app takes the output of a first app, the second app will automatically begin running once the first app has finished.
Connection of outputs occurs in a top-down direction. You may connect the output of a higher step to the input of a lower one, but not vice versa.
You can rearrange the steps in a workflow by dragging steps up and down. Mouse over the background of a stage you would like to drag. Click and hold to drag the step up or down. This will move other apps up or down as needed to accommodate the new position of the app you are moving.
To delete a step, move your mouse over the blue area for the step you would like to delete. This will cause an Actions context menu to appear in the upper right corner. Click it and chose Remove to delete the step.
Once all of the inputs for all of the apps in the workflow have been satisfied (indicated by the black app box showing a green runnable label instead of orange set inputs), you can run the workflow by clicking on the green Start Analysis button in the upper right of the workflow.
Upon completion, a folder will be created with the same name as the workflow in which all data produced by the workflow will be saved. For this reason, it makes sense to name workflows intelligibly. Rename a workflow while in edit mode by clicking on its name at the top of the page.
To edit an existing workflow, from the project's Manage tab, click the checkbox next to the workflow and then click the Edit icon on the right of the toolbar. Alternatively, you can click on the vertical ellipsis affordance on the far right and choose the edit option from the dropdown menu.
