Data Files Used by the Data Model Loader
Phenotypic data must be delivered in a data model defined by the following files:
- A data dictionary CSV file, named
data_dictionary.csvin this example.
- A data CSV file,
- A codings CSV file,
Description: The data dictionary file describes data in the data file (*.data.csv) by file name and by column within the specified data file. The data dictionary lists the metadata for the input
data.csvfile and respective individual columns within the data file. This metadata includes human-readable field names, data types, data coding schemes, and the relationships between the data tables. The information supplied in the data dictionary guides the ingestion process.
data_dictionary.csvfile is similar in structure to a spreadsheet, where columns are defined in Table 1 in the Appendix below. Each row in the data dictionary file represents a field for which an entity, i.e. "patient", "sample", or "encounter," will have a value.
Note: all columns have to be present, however they can be left empty if there are no values for the corresponding entity:field combination. The example below does not include all columns, just enough to illustrate the general format. A complete template can be provided upon request.
Description: The optional entity dictionary file describes entities that the data file represent (*.data.csv). The entity dictionary lists the metadata for each input
data.csv. This metadata includes human-readable entity title, singular and plural labels, and a description. The information supplied in the entity dictionary provides additional contextualized information for the user.
entity_dictionary.csvfile is similar in structure to a spreadsheet, where columns are defined in Table 3 in the Appendix below. Each row in the entity dictionary file represents an entity for which data file and entries in the
All columns have to be present, however they can be left empty if there are no values for the corresponding entity:field combination. The example below does not include all columns, just enough to illustrate the general format. A complete template can be provided upon request.
Description: Phenotypic data must be supplied in a CSV file.
data.csvfile is described in the data dictionary with its name. The
data.csvfile must be a flattened data structure (a table) with a header, columns and rows. The values must be of the following listed types: date, datetime, integer, float, or string.
codings.csvfile is a single file that contains information for coded values present in all the *.data.csv files. Often a meaning is designed for human consumption and codes are designed to optimize data storage and retrieval. For example, instead of storing the value “Acquired pure red cell aplasia [erythroblastopenia]” multiple times in a database, it is easier to code this as the shortened ICD10 code, “D60.” Data are stored as codes in the encoded fields. When retrieving a coded value, such as the ICD10 code "D60", from a database and displaying the value for consumption, the system converts this back to the full meaning, “Acquired pure red cell aplasia [erythroblastopenia],” so that the information is intelligible to the users.
Hierarchical Data: The codings file supports specification of a hierarchy. For example, ICD10 codes, A00 to B99, all point to the category, “Certain infectious and parasitic diseases”. The code, A31, points to “Other sepsis” and codes, A31.X, point to specific types of sepsis. Such a hierarchical structure can be defined in the
codings.csvin the parent_code column (see example below).
codings.csvfile must have columns as defined in Table 2 of the Appendix. Each categorical field in your data will need to have an associated entry in the
codings.csvfile. The same coding_name value must be used in the respective
data_dictionary.csvfield to match each field with its respective codes.
* A value is required unless specified as "or left empty."