Smart Reuse (Job Reuse)

Speed workflow development and reduce testing costs by reusing computational outputs.

A license is required to access the Smart Reuse feature. Contact DNAnexus Sales for more information.

DNAnexus allows organizations to optionally reuse outputs of jobs that share the same executable and input IDs, even if these outputs are across projects or entire organizations. This feature has two primary use cases.

Example Use Cases

Dramatically Speed Up R&D of Workflows

For example, suppose you are developing a workflow, and at each stage, you end up debugging an issue. Let's assume that each stage takes about one hour to develop and run. If you do not reuse outputs as you are developing, the development process takes 1 + 2 + 3 + ... + n hours since at every stage you fix something, you have to recompute results from previous stages you were working on. On the other hand, if you simply reuse results for stages that have matured and are no longer modified, your total development time is now just the total amount of time it takes to develop and run the pipeline (in this case n hours). This is an order of magnitude difference in development time, and the improvement becomes more pronounced for longer workflows.

This feature is also powerful for saving time developing forks of existing workflows. For example, suppose you are a developer in an R&D organization and want to modify the last couple of stages of a production workflow in another organization. As long as the new workflow uses the same executable IDs for the stages before it, the time required for R&D of the forked version is only that of last stages.

Dramatically Reduce Costs When Testing at Scale

In production environments, it is important to test R&D modifications to a workflow at scale. This is especially relevant for workflows used in clinical tests. For example, suppose you are testing a workflow like the forked workflow discussed in the example above. This is a clinical workflow that needs to be tested on thousands of samples (let that number be represented by m) before being vetted to run in a production environment. Let's also suppose the whole workflow takes n hours but you only have modified the last k stages. You save (n-k)m total compute hours. This can add up to dramatic cost savings as m grows and if k is small.

Example Reuse with WDL

To show Smart Reuse, we will use WDL syntax as supported by DNAnexus through our toolkit and by dxCompiler.

task dupfile {
    File infile

    command { cat ${infile} ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

task headfile {
    File infile

    command { head -10 ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

workflow basic_reuse {
    File infile
    call dupfile { input: infile=infile }
    call headfile { input: infile=dupfile.outfile }
}

The workflow above is a two-step workflow that simply duplicates a file and takes the first 10 lines from the duplicate.

Now suppose the user has run the workflow above on some file and simply wants to tweak headfile to output the first 15 lines instead:

task dupfile {
    File infile

    command { cat ${infile} ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

task headfile2 {
    File infile

    command { head -15 ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

workflow basic_reuse_tweaked {
    File infile
    call dupfile { input: infile=infile }
    call headfile { input: infile=dupfile.outfile }
}

Here the only difference is that we renamed headfile, basic_reuse, and changed 10 to 15. The compilation process automatically detects that dupfile is the same but there is a different second stage. The generated workflow therefore uses the original executable ID for dupfile but a different executable ID for headfile2.

When executing basic_reuse_tweaked on the same input file with Smart Reuse enabled, the results from dupfile task are reused. This is because since there is already a job on the DNAnexus Platform that has run that specific executable with the same input file, the system can reuse that file.

When using Smart Reuse with complex WDL workflows involving WDL expressions in input arguments, scatters, and nested sub-workflows, we recommend launching workflows using the --preserve-job-outputs option, to preserve the outputs of all the jobs in the execution tree in the project, and also increase the potential for subsequent Smart Reuse.

Specific Properties

Smart Reuse applies under the following conditions:

• It is available only for jobs run in projects billed to organizations with Smart Reuse enabled.

• It applies only to jobs completed after the organization's policies have been updated to enable Smart Reuse.

Jobs can reuse results from previous jobs if all these criteria are met:

• An existing job exists that used the exact same executable and input IDs (including the function called within the applet). If an input is watermarked, both the watermark and its version must match. Other settings, such as the instance type, do not affect reuse.

• If ignoreReuse: true is set, the job will not be eligible for future reuse.

• The job being reused must have all outputs available and accessible at the time of reuse. If any output is missing or inaccessible, reuse is impossible.

• Each reused job includes an outputReusedFrom field, which points to the original job ID that produced the outputs. This field never refers to another reused job.

• Results can be reused across projects only if the application's dxapp.json file includes "allProjects": "VIEW" in the "access" field.

• You must have at least VIEW access to the original job's outputs, and those outputs must still exist on the Platform. Outputs that have been deleted cannot be reused.

• Reused jobs are reported as having run for 0 seconds and are billed at $0.

• Outputs are assumed to be deterministic.

• If the reused job or workflow is in a different project or folder, the output data is not cloned to the new project or destination folder, since the job or workflow is not actually rerun.

Enable/Disable Smart Reuse

If you are an administrator of a licensed org and want to enable Smart Reuse, run this command:

dx api org-myorg update '{"policies":{"jobReuse":true}}'

If you plan to reuse this feature across projects, you must modify all applet and app configurations with the "allProjects": "VIEW" as described above.

Conversely, set the value to false to disable it. If you are a licensed customer and cannot run the command above, contact DNAnexus Support. If you are interested in this feature and are not a licensed customer, reach out to DNAnexus Sales or your account executive for more information.