Archiving Files
Learn how to archive files, a cost-effective way to retain files in accord with data-retention policies, while keeping them secure and accessible, and preserving file provenance and metadata.
A license is required to use the DNAnexus Archive Service. Contact DNAnexus Sales for more information.
The archiving feature is file-based. Users can also archive individual files, folders, or entire projects and save on storage costs. Users can also easily unarchive one or more files, folders, or projects when they need to make the data available for further analyses.
The DNAnexus Archive Service is currently available via the application program interface (API) in AWS and Microsoft Azure regions.
To understand the archival life cycle as well as which operations can be performed on files and how billing works, it’s helpful to understand the different file states associated with archival. A file in a project can assume one of four archival states:
Archival states | Details |
live | The file is in standard storage, such as AWS S3 or Azure Blob. |
archival | Archival requested on the current file, but other copies of the same file are in the live state in multiple projects with the same billTo entity. The file is still in standard storage. |
archived | The file is in archival storage, such as AWS S3 Glacier or Azure Blob ARCHIVE. |
unarchiving | Unarchival requested on the current file. The file is in transition from archival storage to standard storage. |
Different states of a file allow different operations to the file. See the table below, for which operations can be performed based on a file’s current archival state.
Archival states | Download | Clone | Compute | Archive | Unarchive |
live | Yes | Yes | Yes | Yes | No |
archival | No | Yes* | No | No | Yes (Cancel archive) |
archived | No | Yes | No | No | Yes |
unarchiving | No | No | No | No | No |
* Clone operation would fail if the object is actively transitioning from
archival to archived.When the
project-xxxx/archive API is called upon a file object, the file transitions from the live state to the archival state. Only when all copies of a file in all projects with the same billTo organization are in the archival state, does the file transition to the archived state automatically by the platform.Likewise, when the
project-xxxx/unarchive API is called upon a file in the archived state, the file transitions from the archived to the unarchiving state. During the unarchiving state, the file is being restored by the third-party storage platform (e.g., AWS or Azure). The unarchiving process may take a while depending on the retrieval option selected for the specific platform. Finally, when the unarchival process is completed, and the file becomes available on standard storage, the file is transitioned to a live state.
The File-based Archive Service allows users who have the
CONTRIBUTE or ADMINISTER permissions to a project to archive or unarchive files that reside in the project. Via API calls, users can archive or unarchive files, folders, or entire projects, although the archival process itself happens at the file level. The API can accept a list of up to 1000 files for archival and unarchival. When archiving or unarchiving folders or projects, the API by default will archive or unarchive all the files at the root level and those in the subfolders recursively. If you archive a folder or a project that includes filess in different states, the Service will only archive files that are in the live state and skip files that are in other states. Likewise, if you unarchive a folder or a project that includes files in different states, the Service will only unarchive files that are in the archived state, transition archival files back to the live state, skip files in other states.All the fees associated with the archival process of a file get billed to the billTo organization of the project. There are several charges associated with the archival:
Standard storage charge: The monthly storage charge for files that are located in the standard storage on the platform. The files in the
live and archival state incur this charge. The archival state indicates that the file is waiting to be archived or that other copies of the same file in other projects are still in the live state, so the file is in standard storage (such as AWS S3). The standard storage charge continues to get billed until all copies of the file are requested to be archived and eventually the file is moved to archival storage and transitioned into the archived state.Archival storage charge: The monthly storage charge for files that are located in archival storage on the platform. Files in the
archived state incur a monthly archival storage charge.Retrieval fee: The retrieval fee is a one-time charge at the time of unarchival based on the data volume being unarchived. Retrieval fees for third-party services can be found at:
Early retrieval fee: Because the Archive Service is designed for long-term storage of data that are infrequently used, there is necessarily a retrieval fee associated with data that are retrieved before these long-term storage periods have been met. For AWS regions, this time period is 90 days, and for Microsoft Azure regions, this time period is 180 days. Data that are unarchived less than the minimum requirement days incur a pro-rated early retrieval charge, which is equal to the archival charge for the remaining days.
When using the Archive Service, we recommend the following best practices.
- The Archive Service does not work on sponsored projects. If you want to archive files within a sponsored project, then you must move files into a different project or end the project sponsorship before archival.
- If a file is shared in multiple projects, archiving one copy in one of the projects will only transition the file into the
archivalstate, which still incurs the standard storage cost. To achieve the lower archival storage cost, you need to ensure that all copies of the file in all projects with the samebillToorg are being archived. When all the copies of the file transition into thearchivalstate, the Service automatically transitions the files from thearchivalstate to thearchivedstate. We recommend using theallCopiesoption of the API to force archiving all the copies of the file. You must be the org ADMIN of thebillToorg of the current project to use theallCopiesoption.Refer to the following example: Thefile-xxxxhas copies inproject-xxxx,project-yyyy, andproject-zzzzwhich are sharing the samebillToorg (org-xxxx). You are theADMINISTERofproject-xxxx, and aCONTRIBUTEofproject-yyyy, but do not have any role inproject-zzzz. You are the org ADMIN of the projectbillToorg, and try to archive all copies of files in all projects with the samebillToorg using /project-xxxx/archive:- 1.List all the copies of the file in the
org-xxxx$ dx api file-xxxx listProjects '{"archivalInfoForOrg":"org-xxxx"}'{"project-xxxx": "ADMINISTER","project-yyyy": "CONTRIBUTE","liveProjects": ["project-xxxx","project-yyyy","project-zzzz"]} - 2.Force archiving all the copies of
file-xxxx$ dx api project-xxxx archive '{"files": ["file-xxxx"], "allCopies": true}'{"id": "project-xxxx""count": 1} - 3.All copies of
file-xxxxwill be archived and transitioned into thearchivedstate.
Last modified 11mo ago