Archiving files provides users with a cost-effective way to store files according to their data retention policies while maintaining security, file provenance, meta-data searchability, and other core features of the DNAnexus platform. Users can archive individual files, folders, or entire projects and save on storage costs. Users can also easily unarchive one or more files, folders, or projects when they need to make the data available for further analyses.

DNAnexus Archival Service is currently available via the application program interface (API) in AWS and Microsoft Azure regions.

This page provides an overview of the DNAnexus Archival Service.

NOTE: A license is required to access the archival service. Please contact sales@dnanexus.com for more information.

Overview of the Archival Service

File Archival States

To understand the archival life cycle as well as which operations can be performed on files and how billing works, it’s helpful to understand the different file states associated with archival. A file in a project can assume one of four archival states:

Different states of a file allow different operations to the file. See the table below, for which operations can be performed based on a file’s current archival state.

* Clone operation would fail if the object is actively transitioning from archival to archived.

File Archival Life Cycle

When the project-xxxx/archive API is called upon a file object, the file transitions from the live state to the archival state. Only when all copies of a file in all projects with the same billTo organization are in the archival state, does the file transition to the archived state automatically by the platform.

Likewise, when the project-xxxx/unarchive API is called upon a file in the archived state, the file transitions from the archived to the unarchiving state. During the unarchiving state, the file is being restored by the third-party storage platform (e.g., AWS or Azure). The unarchiving process may take a while depending on the retrieval option selected for the specific platform. Finally, when the unarchival process is completed, and the file becomes available on standard storage, the file is transitioned to a live state.

Operations of the Archival Service

The File-based Archival Service allows users who have the CONTRIBUTE or ADMINISTER permissions to a project to archive or unarchive files that reside in the project. Via API calls, users can archive or unarchive files, folders, or entire projects, although the archival process itself happens at the file level. The API can accept a list of up to 1000 files for archival and unarchival. When archiving or unarchiving folders or projects, the API by default will archive or unarchive all the files at the root level and those in the subfolders recursively. If you archive a folder or a project that includes filess in different states, the service will only archive files that are in the live state and skip files that are in other states. Likewise, if you unarchive a folder or a project that includes files in different states, the service will only unarchive files that are in the archived state, transition archival files back to the live state, skip files in other states.

Archival Billing

All the fees associated with the archival process of a file get billed to the billTo organization of the project. There are several charges associated with the archival:

Standard storage charge: The monthly storage charge for files that are located in the standard storage on the platform. The files in the live and archival state incur this charge. The archival state indicates that the file is waiting to be archived or that other copies of the same file in other projects are still in the live state, so the file is in standard storage (such as AWS S3). The standard storage charge continues to get billed until all copies of the file are requested to be archived and eventually the file is moved to archival storage and transitioned into the archived state.

Archival storage charge: The monthly storage charge for files that are located in archival storage on the platform. Files in the archived state incur a monthly archival storage charge.

Retrieval fee: The retrieval fee is a one-time charge at the time of unarchival based on the data volume being unarchived. Retrieval fees for third-party services can be found at:

• Amazon AWS: https://aws.amazon.com/glacier/pricing​

• Microsoft Azure: https://azure.microsoft.com/en-us/pricing/details/storage/blobs​

Early retrieval fee: Because the archival service is designed for long-term storage of data that are infrequently used, there is necessarily a retrieval fee associated with data that are retrieved before these long-term storage periods have been met. For AWS regions, this time period is 90 days, and for Microsoft Azure regions, this time period is 180 days. Data that are unarchived less than the minimum requirement days incur a pro-rated early retrieval charge, which is equal to the archival charge for the remaining days.

Archival Best Practices

When using the Archival service, we recommend the following best practices.

• Archival service does not work on sponsored projects. If you want to archive files within a sponsored project, then you must move files into a different project or end the project sponsorship before archival.

• If a file is shared in multiple projects, archiving one copy in one of the projects will only transition the file into the archival state, which still incurs the standard storage cost. To achieve the lower archival storage cost, you need to ensure that all copies of the file in all projects with the same billTo org are being archived. When all the copies of the file transition into the archival state, the service automatically transitions the files from the archival state to the archived state. We recommend using the allCopies option of the API to force archiving all the copies of the file. You must be the org ADMIN of the billTo org of the current project to use the allCopies option.

  Refer to the following example: The file-xxxx has copies in project-xxxx, project-yyyy, and project-zzzz which are sharing the same billTo org (org-xxxx). You are the ADMINISTER of project-xxxx, and a CONTRIBUTE of project-yyyy, but do not have any role in project-zzzz. You are the org ADMIN of the project billTo org, and try to archive all copies of files in all projects with the same billTo org using /project-xxxx/archive:

  1. List all the copies of the file in the org-xxxx

     $ dx api file-xxxx listProjects '{"archivalInfoForOrg":"org-xxxx"}'
     {
     "project-xxxx": "ADMINISTER",
     "project-yyyy": "CONTRIBUTE",
     "liveProjects": [
      "project-xxxx",
      "project-yyyy",
      "project-zzzz"
     ]
     }

  2. Force archiving all the copies of file-xxxx

     $ dx api project-xxxx archive '{"files": ["file-xxxx"], "allCopies": true}'
     {
     "id": "project-xxxx"
     "count": 1
     }

  3. All copies of file-xxxx will be archived and transitioned into the archived state.