gms | German Medical Science

Introduction: When processing clinical text, often captured via semi- or unstructured free text fields, Named Entity Recognition (NER, [1]) is essential for the realization of Information Extraction (IE) or Information Retrieval (IR) systems [2]. Nouns and named entities (NE) often act as the most relevant terms for matching, for instance in a database or against an index.

Frei and Kramer presented GERNERMED [3], a model for detecting specific medical NEs such as: (prescribed) drugs, strength, route, form, dosage, frequency, and duration. However, to the best of the author’s knowledge, no broad, non-proprietary NER model has been published that can also detect anatomic (body) structures, symptoms, and diseases as covered by the UMLS [4].

This study demonstrates the feasibility of training an open, medical NER model and evaluates it against real-world, clinical text material.

Methods: For the training phase, a synthetic text corpus was compiled, comprising of close to 88.000 health-related Wikipedia articles and its full texts, dating July 2023. The corpus contained ~2.4mio sentences. Next, nouns and noun phrases in the training corpus were automatically tagged against a UMLS concept list; it included ~56.600 medical NEs translated to German. Technically, the training was conducted with the open-source framework Apache OpenNLP [5], in version 2.3.3. For training a Maximum Entropy NER model, the parameters were chosen as follows:

?????training.algorithm=maxent;training.iterations=150;training.cutoff=3;training.threads=8;language=de;use.token.end=false

The resulting model file was then persisted for re-use in NLP applications to detect named (medical) entities.

For the performance evaluation of the DE-NERmed model, n=101 text fragments were randomly selected from discharge letters, originally created in the Chest Pain Unit at the Heidelberg University Hospital. For inclusion, a text fragment had to consist of at least 20 tokens. After????? preparation of the evaluation corpus, both F₁ score, and Accuracy were computed.

Results: Training of the 'DE-NERmed-Wiki_2023-maxent' model was carried out on June 25, 2024 in a cluster environment at the Faculty of Computer Science at Heilbronn University. In its binary form, it requires ~5 GB of RAM at runtime.

The model achieved: F₁=0.8761 and an Accuracy of 0.8922 (TP=905; TN=1214; FP=65; FN=191). It detected most of the relevant medical NEs, associated with the cardiology and the general medical domain. Misclassifications occurred primarily for NEs which were representative for both, the general and the medical language.

The DE-NERmed model, demo code with text examples, and accompanying data is made available at: https://github.com/mawiesne/DE-NERmed

Discussion & conclusion: The topical variety, the linguistic properties and annotations, and the recency of the Wikipedia material could not be controlled for this study, due to the volume of the training corpus.

The evaluation sample was restricted to 101 text fragments. Nonetheless, those were sampled from unstructured text material which originated from real-world, clinical text material. The construction of a more extensive NER evaluation corpus offers future research potential.

The technical feasibility of training and practical use of an NER model for the German (bio)medical domain was demonstrated. The pre-annotated training corpus and the binary DE-NERmed model file are contributed for scientific comparisons, or for practical use in NLP software pipelines.

Competing interests: Martin Wiesner is committer and member of the Project Management Committee (PMC) in the OpenNLP project of the Apache Software Foundation.

The authors declare that an ethics committee vote is not required.