Smart Reuse (Job Reuse)

Speed workflow development and reduce testing costs by reusing computational outputs.

A license is required to access the Smart Reuse feature. Contact DNAnexus Sales for more information.

DNAnexus allows organizations to optionally reuse outputs of jobs that share the same executable and input IDs, even if these outputs are across projects or entire organizations. This feature has two primary use cases.

Example Use Cases

Dramatically Speed Up R&D of Workflows

For example, suppose you are developing a workflow, and at each stage you end up debugging an issue. Each stage takes about one hour to develop and run. If you do not reuse outputs during development, the process takes 1 + 2 + 3 + ... + n hours because at every stage you fix something and must recompute results from previous stages. By reusing results for stages that have matured and are no longer modified, the total development time equals the time it takes to develop and run the pipeline (in this case n hours). This is an order-of-magnitude reduction in development time, and the improvement becomes more pronounced for longer workflows.

This feature also saves time when developing forks of existing workflows. For example, suppose you are a developer in an R&D organization and want to modify the last couple of stages of a production workflow in another organization. As long as the new workflow uses the same executable IDs for the earlier stages, the time required for R&D of the forked version equals the time for the last stages.

Dramatically Reduce Costs When Testing at Scale

In production environments, test R&D modifications to a workflow at scale. This is especially relevant for workflows used in clinical tests. For example, suppose you are testing a workflow like the forked workflow discussed earlier. This clinical workflow must be tested on thousands of samples (let that number be represented by m) before it is vetted for production. Suppose the whole workflow takes n hours but only the last k stages changed. You save (n-k)m total compute hours. This can add up to dramatic cost savings as m grows and if k is small.

Example Reuse with WDL

To show Smart Reuse, the following example uses WDL syntax as supported by DNAnexus SDK and dxCompiler.

task dupfile {
    File infile

    command { cat ${infile} ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

task headfile {
    File infile

    command { head -10 ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

workflow basic_reuse {
    File infile
    call dupfile { input: infile=infile }
    call headfile { input: infile=dupfile.outfile }
}

The workflow above is a two-step workflow that duplicates a file and takes the first 10 lines from the duplicate.

Suppose the user has run the workflow above on some file and wants to tweak headfile to output the first 15 lines instead:

task dupfile {
    File infile

    command { cat ${infile} ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

task headfile2 {
    File infile

    command { head -15 ${infile} > outfile.txt  }
    output { File outfile = 'outfile.txt' }
}

workflow basic_reuse_tweaked {
    File infile
    call dupfile { input: infile=infile }
    call headfile { input: infile=dupfile.outfile }
}

Here the only differences are the renamed headfile and basic_reuse, and the change from 10 to 15. The compilation process automatically detects that dupfile is the same but the second stage differs. The generated workflow therefore uses the original executable ID for dupfile but a different executable ID for headfile2.

When executing basic_reuse_tweaked on the same input file with Smart Reuse enabled, the results from dupfile task are reused. This is because since there is already a job on the DNAnexus Platform that has run that specific executable with the same input file, the system can reuse that file.

When using Smart Reuse with complex WDL workflows involving WDL expressions in input arguments, scatters, and nested sub-workflows, we recommend launching workflows using the --preserve-job-outputs option. This preserves the outputs of all jobs in the execution tree in the project and increases the potential for subsequent Smart Reuse.

Specific Properties

Smart Reuse applies under the following conditions:

• It is available only for jobs run in projects billed to organizations with Smart Reuse enabled.

• It applies only to jobs completed after the organization's policies have been updated to enable Smart Reuse.

Jobs can reuse results from previous jobs if all these criteria are met:

• An existing job exists that used the exact same executable and input IDs (including the function called within the applet). If an input is watermarked, both the watermark and its version must match. Other settings, such as the instance type, do not affect reuse.

• If ignoreReuse: true is set, the job is not eligible for future reuse.

• The job being reused must have all outputs available and accessible at the time of reuse. If any output is missing or inaccessible, reuse is impossible.

• Each reused job includes an outputReusedFrom field, which points to the original job ID that produced the outputs. This field never refers to another reused job.

• Results can be reused across projects only if the application's dxapp.json file includes "allProjects": "VIEW" in the "access" field.

• You must have at least VIEW access to the original job's outputs, and those outputs must still exist on the Platform. Outputs that have been deleted cannot be reused.

• Reused jobs are reported as having run for 0 seconds and are billed at $0.

• Outputs are assumed to be deterministic.

• If the reused job or workflow is in a different project or folder, the output data is not cloned to the new project or destination folder, since the job or workflow is not actually rerun.

Enable/Disable Smart Reuse

If you are an administrator of a licensed org and want to enable Smart Reuse, run this command:

dx api org-myorg update '{"policies":{"jobReuse":true}}'

If you plan to reuse this feature across projects, you must modify all applet and app configurations with the "allProjects": "VIEW" as described above.

Conversely, set the value to false to disable it. If you are a licensed customer and cannot run the command above, contact DNAnexus Support. If you are interested in this feature and are not a licensed customer, reach out to DNAnexus Sales or your account executive for more information.