Ingestion Data Types
Data Model Loader support a number of field types during ingestion of raw files. The following field types are supported with the outlined configurations. Note that during ingestion, all values in a field must conform to the type format supported and indicated. This includes special values used to indicate "Sparse" values.
For multi-select fields, the field in the raw data csv would be entered as "<value 1>|<value 2>|<value n>" with a "|" delimiter for entries where multiple values selected. Note that not all entries require multiple entries for a field to be treated as multi-select. The "value n", part of the multi-select must be non-null / non-empty.
A date (i.e. timepoint with day granularity) without special values (other than NULL). An example is Date of Birth
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
date | | | | |
The following formats are supported for Date
- yyyy
- yyyy-[m]m
- yyyy-[m]m-[d]d
A date with a few special values (other than NULL). An example is Date of Diagnosis where a date of "1900-01-01" indicates "Forgot to ask question".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
date | <provided> | yes | | |
A datetime without special values (other than NULL). An example is datetime of hospital admission. Datetimes are stored in the system as UTC.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
datetime | | | | |
The following formats are supported for Datetime
- yyyy
- yyyy-[m]m
- yyyy-[m]m-[d]d
- yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]
- yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]Z
- yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]-[h]h:[m]m
- yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]+[h]h:[m]m
- yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]
- yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]Z
- yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]-[h]h:[m]m
- yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us]+[h]h:[m]m
A datetime (i.e. timepoint with microsecond granularity) with a few special values (other than NULL). An example is Datetime of Treatment where a date of "1900-01-01 01:00" indicates "NA".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
datetime | <provided> | yes | | |
A double-precision floating point measurement without special values (other than NULL). An example is Weight.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
float | | | | |
The following formats are supported for Float
- #.#
- #.#E#
- -#.#E#
A double-precision floating point measurement with a few special values (other than NULL). An example is Blood Glucose Level where a value of "-1.0" indicates "Read Failure".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
float | <provided> | yes | | |
An integer measurement without special values (other than NULL). An example is Age.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | | | | |
The following formats are supported for Integer
- #
An integer measurement with a few special values (other than NULL). An example is Cups of Coffee Consumed where a value of "-1" indicates "Prefer not to answer".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | <provided> | yes | | |
A single-choice value (string) from a controlled vocabulary of possible options (or NULL), originally encoded as integer. An example is Have you traveled in the last 30 days? where a value of "1" means "Yes" and a value of 0 means "No"
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | <provided> | | | |
A single-choice value (string) from a controlled vocabulary of possible options (or NULL) that are organized in a hierarchy, originally encoded as integer. An example is Job Code At Visit where an integer of "12" means "VP of Operations" which is a child of "Corporate Managers And Senior Officials".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | <provided> | | | <provided for child values> |
A multiple-choice value (string) from a controlled vocabulary of possible options (or NULL), originally encoded as integer. An example is Patient Ethnic Background where 0 means "African", 1 means "Caucasian", 2 means "Asian"... etc, and a patient may be part of one or many races.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | <provided> | | yes | |
A multiple-choice value (string) from a controlled vocabulary of possible options (or NULL) that are organized in a hierarchy, originally encoded as integer. An example is Cancer Codes Self Reported where 0 means "Parotid Gland Cancer", 2 means "Other Salivary Gland Cancer" and both have a parent of 3 "Salivary Gland Cancer" .. etc, and a patient may have one or more cancer codes entered.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
integer | <provided> | | yes | <provided for child values> |
A free-form string without special values (other than NULL). An example is UUID or Special Instructions.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | | | | |
The following formats are supported for String
- any string
- for quote escaping use "" (e.g. use <"the ""quote"" phrase"> to get <the "quote" phrase>)
A single-choice value (string) from a controlled vocabulary of possible options (or NULL), originally encoded as string. An example is T-Shirt Size where "S" means "Small"
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | | | |
A single-choice value (string) from a controlled vocabulary of possible options (or NULL) that are organized in a hierarchy, originally encoded as string. An example is State of Residency where a value of "CA" means "California" and a value of "OR" means "Oregon" and both values have a parent of "US" that means "United States of America".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | | | <provided for child values> |
A multiple-choice value (string) from a controlled vocabulary of possible options (or NULL), originally encoded as string. An example is Primary Diagnosis where values are encoded to match ICD10 diagnostic codes.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | | yes | |
A multiple-choice value (string) from a controlled vocabulary of possible options (or NULL) that are organized in a hierarchy, originally encoded as string. An example is Secondary Diagnosis where an entry may have one or more values encoded to match .
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | | yes | <provided for child values> |
A single-choice value (string) from a controlled vocabulary of possible options (originally encoded as string) or a write-in value (or NULL). Most values are expected to be from the controlled vocabulary. An example is Prescribed Medication where each entry is a single item from a list of RXNORM medications where "RX5640" means "Ibuprofen", etc. but a value may be written in if the medication is not available as an option.
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | yes | | |
A single-choice value (string) from a controlled vocabulary of possible options that are organized in a hierarchy (originally encoded as string) or a write-in value (or NULL). Most values are expected to be from the controlled vocabulary. An example is Previous Diagnostic Codes ICD-10 where the code "D03" has meaning "Melanoma in situ", and the code "D03.0" has meaning "D03.0 Melanoma in situ of lip", and "D03.0" is hierarchically situated under "D03", or any other free-form string that is not part of ICD10 like "Other trauma".
Ingestion files are configured as follows:
type | coding_name | is_sparse_coding | is_multi_select | (coding.csv) parent_code |
string | <provided> | yes | | <provided for child values> |
Last modified 2mo ago