Field Types (config_type reference) =================================== Each field in a parse configuration has a ``config_type`` that controls how the parser populates it. The behaviour of each type is implemented in ``data_driven_parse.py`` by a corresponding ``do_*`` function. None ---- *Implemented by:* ``do_none_fields()`` Set fields whose config_type is None to None in output_dict. CONSTANT -------- *Implemented by:* ``do_constant_fields()`` Write CONSTANT fields from config into output_dict, truncating strings to the allowed length. FILENAME -------- *Implemented by:* ``do_filename_fields()`` Write the source filename into any FILENAME-typed fields in output_dict. FIELD ----- *Implemented by:* ``do_basic_fields()`` Extract FIELD and PK values from the XML element and write them into output_dict. PK values are also appended to pk_dict so downstream FK fields can reference them. String values are whitespace-normalized and truncated to the configured max length. FK -- *Implemented by:* ``do_foreign_key_fields()`` When a configuration has an FK field, it uses the tag in that configuration to find corresponding values from PK fields. This mechanism is intended for PKs uniquely identified in a CCDA document header for any places in the sections it would be used as an FK. This is typically true for person_id and visit_occurrence_id, but there are exceptions. In particular, some documents have multiple encounters, so you can't just naively choose the only visit_id because there are many. Choosing the visit is more complicated, because it requires a join (on date ranges) between the domain table and the encounters table, or portion of the header that has encompassingEncounters in it. This code, the do_foreign_key_fields() function operates in too narrow a context for that join. These functions are scoped down to processing a single config entry for a particular OMOP domain. The output_dict, parameter is just for that one domain. It wouldn't include the encounters. For example, the measurement_results.py file has a configuration for parsing OMOP measurement rows out of an XML file. The visit.py would have been previosly processed and it's rows stashed away elsewhere in the parse_doc() function whose scope is large enough to consider all the configurations. So the visit choice/reconciliation must happen from there. TL;DR not all foreign keys are resolved here. In particular, domain FK references, visit_occurrence_id, in cases where more than a single encounter has previously been parsed, are not, can not, be resolved here. See the parse_doc() function for how it is handled there. DERIVED ------- *Implemented by:* ``do_derived_fields()`` Do/compute derived values now that their inputs should be available in the output_dict Except for a special argument named 'default', when the value is what is other wise the field to look up in the output dict. This set-up is for functions that expect explicit named arguments. This code here adds values for those arguments to the the dictionary passed to the function. It's tempting to want to pass a list of arguments, but that's not how this function works. Also a PK DERIVED2 -------- *Implemented by:* ``do_derived2_fields()`` Compute DERIVED2 fields using functions that receive the full output_dict and argument_list. Unlike DERIVED, the called function is responsible for fetching its own values from output_dict using the key_list in argument_list, allowing a variable number of inputs. HASH ---- *Implemented by:* ``do_hash_fields()`` Compute HASH fields by hashing a list of named input fields into a single ID. Similar to DERIVED but takes a list of field names rather than individually named arguments. The resulting hash is also stored in pk_dict so it can be used as a PK/FK reference. Note: hash IDs are 64-bit but OMOP integer columns are typically 32-bit — use with care. See the code for data_type-based conversion logic. where a different kind of hash is beat into an integer. ALSO A PK PRIORITY -------- *Implemented by:* ``do_priority_fields()`` ARGS expected in config: 'config_type': 'PRIORITY', 'defult': 0, in case there is no non-null value in the priority change and we don't want a null value in the end. 'order': 17 Returns the list of priority_names so the chosen one (first non-null) can be added to output fields Also, adds this field to the PK list? This is basically what SQL calls a coalesce. Within the config_dict, find all fields tagged with priority and group them by their priority names in a dictionary keyed by that name Ex. { 'person_id': [ ('person_id_ssn', 1), ('person_id_unknown', 2) ] Sort them, choose the first one that is not None. NB now there is a separate config_type PRIORITY to compliment the priority attribute. So you might have person_id_npi, person_id_ssn and person_id_hash tagged with priority attributes to create a field person_id, but then also another field, just plain person_id. The point of it is to have a unique place to put that field's order attribute. The code here (and in the ordering code later) must be aware of a that field in the config_dict (where it isn't used) ...and not clobber it. It's an issue over in the sorting/ordering.