gms | German Medical Science

Introduction: Semantic annotation of electronic health records (EHR) in a structured way improves interoperability in patient care and research. The Clinical Data Interchange Standards Consortium (CDISC) Operational Data Model (ODM) XML standard allows syntactic structuring and semantic annotation of metadata. Whereas other standards such as the Fast healthcare interchange resources (FHIR) play an important role in clinically, CDISC ODM is widely used in clinical research [1]. EHR data from intensive care unit (ICU) settings typically include large parameter sets to capture characteristics of complex diseases. Manual annotation with controlled vocabularies is difficult and time-consuming. To automate semantic annotation, transformer-based natural language processing (NLP) models are promising due to representation of contextual medical knowledge within the model.

State of the art: Current approaches to EHR annotation include manual, semi-automated or rule-based methods and transformer-based models for free text annotation. A semi-automated approach for the re-use of annotations [2] was described in the context of a pragmatic metadata repository (MDR). The pragmatic MDR divides ODMs into atomic elements, aggregating similar types of elements and allows frequency-based search for United Medical Language System (UMLS) concepts. The transformer model Medical Annotation tool (MedCat) provides an automated annotation approach trained on large amounts of annotated medical free texts [3].

Concept: We will convert unstructured parameter sets from the Amsterdam University Medical Center Database Heidelberg University Hospital and the Bern University Hospital and Swiss Federal Institute of Technology’s (ETH) HiRID dataset [4] as well as the ICU parameters of MIMIC III from comma-separated values format into structured CDISC ODM, and harmonise languages into English through a transformer based approach.

Semantic annotation will be performed using the rule-based annotation of the pragmatic MDR [2], large transformer models re-trained on named entity recognition and linking (NER+L), and a subset annotated by medical practitioners as a gold standard.

Re-tenraining of the transformers will be evaluated in standardised manner by using the precision, recall and F1 metrics commonly used in the evaluation of NLP tasks.

To compare the performance of the entity linking (EL), Krippendorff's alpha will be used to measure semantic distances.

Implementation: We developed a java-based conversion tool that can implement the syntactic schema of the CDISC ODM standard using semi-structured data in list form. The transformer models MedCat [3] and GPT Neo-XT base 20B will be re-trained on data from a large, fully annotated MDR and data from the Medical Information Mart for Intensive Care (MIMIC)-III database [5]. To ensure a meaningful comparison between the different sites and annotation approaches, core data elements for a clinical use case will be created. A subset of this core dataset will be manually annotated by medical practitioners to create a benchmark. After re-training independent medical practicioners will evaluate annotation performance in a structured manner. Preliminary results for a pilot study of this evaluation will be presented at the conference.

Lessons learned: Manual annotation is time-consuming, limiting the volume of metadata that can be annotated in given time. Using the rule-based approach significantly reduces the required time on task. Re-training the transformer models requires high computing resources exceeding conventional capabilities but promise to improve coding rate and accuracy. As a result, high performance computing infrastructures will be employed.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.