Archiving Files
Learn how to archive files, a cost-effective way to retain files in accord with data-retention policies, while keeping them secure and accessible, and preserving file provenance and metadata.
Last updated
Was this helpful?
Learn how to archive files, a cost-effective way to retain files in accord with data-retention policies, while keeping them secure and accessible, and preserving file provenance and metadata.
Last updated
Was this helpful?
Archiving in DNAnexus is file-based. You can archive individual files, folders with files, or entire projects' files and save on storage costs. You can also easily unarchive one or more files, folders, or projects when you need to make the data available for further analyses.
The DNAnexus Archive Service is available via the API in Amazon AWS and Microsoft Azure regions.
To understand the archival life cycle as well as which operations can be performed on files and how billing works, it’s helpful to understand the different file states associated with archival. A file in a project can assume one of four archival states:
Archival states
Details
live
The file is in standard storage, such as AWS S3 or Azure Blob.
archival
Archival requested on the current file, but other copies of the same file are in the live state in multiple projects with the same billTo entity. The file is still in standard storage.
archived
The file is in archival storage, such as AWS S3 Glacier or Azure Blob ARCHIVE.
unarchiving
Unarchival requested on the current file. The file is in transition from archival storage to standard storage.
Different states of a file allow different operations to the file. See the table below, for which operations can be performed based on a file’s current archival state.
Archival states
Download
Clone
Compute
Archive
Unarchive
live
Yes
Yes
Yes
Yes
No
archival
No
Yes*
No
No
Yes (Cancel archive)
archived
No
Yes
No
No
Yes
unarchiving
No
No
No
No
No
* Clone operation would fail if the object is actively transitioning from archival to archived.
When the project-xxxx/archive API is called upon a file object, the file transitions from the live state to the archival state. Only when all copies of a file in all projects with the same billTo organization are in the archival state, does the file transition to the archived state automatically by the platform.
Likewise, when the project-xxxx/unarchive API is called upon a file in the archived state, the file transitions from the archived to the unarchiving state. During the unarchiving state, the file is being restored by the third-party storage platform (e.g., AWS or Azure). The unarchiving process may take a while depending on the retrieval option selected for the specific platform. Finally, when the unarchival process is completed, and the file becomes available on standard storage, the file is transitioned to a live state.
The File-based Archive Service allows users who have the CONTRIBUTE or ADMINISTER permissions to a project to archive or unarchive files that reside in the project.
Using API, users can archive or unarchive files, folders, or entire projects, although the archival process itself happens at the file level. The API can accept a list of up to 1000 files for archival and unarchival.
When archiving or unarchiving folders or projects, the API by default archives or unarchives all the files at the root level and those in the subfolders recursively. If you archive a folder or a project that includes files in different states, the Service will only archive files that are in the live state and skip files that are in other states. Likewise, if you unarchive a folder or a project that includes files in different states, the Service will only unarchive files that are in the archived state, transition archival files back to the live state, skip files in other states.
All the fees associated with the archival process of a file get billed to the billTo organization of the project. There are several charges associated with the archival:
Standard storage charge: The monthly storage charge for files that are located in the standard storage on the platform. The files in the live and archival state incur this charge. The archival state indicates that the file is waiting to be archived or that other copies of the same file in other projects are still in the live state, so the file is in standard storage, such as AWS S3. The standard storage charge continues to get billed until all copies of the file are requested to be archived and eventually the file is moved to archival storage and transitioned into the archived state.
Archival storage charge: The monthly storage charge for files that are located in archival storage on the platform. Files in the archived state incur a monthly archival storage charge.
Retrieval fee: The retrieval fee is a one-time charge at the time of unarchival based on the volume of data being unarchived.
Early unarchival fee: If you retrieve or delete data from archival storage before the required retention period is met, a dearchival early retrieval fee applies. This is 90 days for AWS regions and 180 days for Microsoft Azure regions. You will be charged a pro-rated fee equivalent to the archival storage charges for any remaining days within that period.
When using the Archive Service, we recommend the following best practices.
If a file is shared in multiple projects, archiving one copy in one of the projects will only transition the file into the archival state, which still incurs the standard storage cost. To achieve the lower archival storage cost, you need to ensure that all copies of the file in all projects with the same billTo org are being archived. When all the copies of the file transition into the archival state, the Service automatically transitions the files from the archival state to the archived state. We recommend using the allCopies option of the API to force archiving all the copies of the file. You must be the org ADMIN of the billTo org of the current project to use the allCopies option.
List all the copies of the file in the org-xxxx .
Force archiving all the copies of file-xxxx .
All copies of file-xxxx will be archived and transitioned into the archived state.
The Archive Service does not work on . If you want to archive files within a sponsored project, then you must move files into a different project or end the project sponsorship before archival.
Refer to the following example: The file-xxxx has copies in project-xxxx, project-yyyy, and project-zzzz which are sharing the same billTo org (org-xxxx). You are the ADMINISTER of project-xxxx, and a CONTRIBUTE of project-yyyy, but do not have any role in project-zzzz. You are the org ADMIN of the project billTo org, and try to archive all copies of files in all projects with the same billTo org using :
