gms | German Medical Science

Introduction: A large part of routine medical documentation is most interesting for a secondary use but still often primarily provided as free text. Besides the interest in providing more primarily structured documentation, there also efforts in extracting information from free texts in order to unlock the already existing data. These efforts concentrate on extracting single parameters that are more or less clearly defined within the text, as e.g. single echocardiographic parameters [1], [2]. Such approaches usually fail if not an individual parameter needs to be detected, but a complex syndrome, e.g. heart failure (HF). The definition of HF depends on the very specific combination of different parameters, which are only selectively provided in discharge letters. Here we describe our effort to extract the HF diagnosis by utilizing our previous echocardiographic information with high accuracy scores extraction and other strategies, e.g. searches in discharge letters [3].

Methods: We utilized the clinical data warehouse (DWH) deployed at our hospital (including data from multiple domains and information systems and an ad-hoc information extraction feature [4]) to select and extract data from 1042 inpatients of the department of internal medicine. A cardiologist with long-standing experience provided a gold standard definition of the HF diagnosis. Eighteen queries were defined using the DWH query interface, with any hit meaning the existence of HF. Each query represents a specific medical concept relevant for the HF diagnosis that was documented as structured data (3 queries, e.g. all HF-related ICD diagnoses), data from information extraction of the echocardiographic report (2 queries, e.g. Left ventricular ejection fraction ≤45) and the existence of texts within the discharge letter including synonyms (13 queries, e.g. existence of “reduced left ventricular function”). An algorithm was defined via permutation testing on all queries to optimize for a high F1-score. Another algorithm was manually defined using pure ICD codes. These scores are compared to parameter extractions provided earlier [1].

Results: 222 patients of the 1042 were labeled by the cardiologist to have heart failure. The ICD-only definition resulted in a precision of 94%, a recall of 50% and an F1-score of 65%. The F1-score optimized algorithm resulted in a precision of 89%, a recall of 84% and a F1-score of 86%. The previous extraction of the 29 single parameters from echocardiography resulted in micro averages of 99% (precision), 99% (recall), and 99% (F1-score) [1].

Discussion: While it is rather simple to extract parameters with high F1-scores from texts that are steady and uniformly documented, it is much harder to use a direct extraction approach to get high F1-scores for medical concepts that are less clearly documented, as shown in this analysis. Literature about HF-identification algorithms usually shows even lower F1-scores than our 86% (e.g. 82% in [5]). One approach that we consider for the future are machine-learning techniques, which might improve F1-scores of medical concepts with a rather fuzzy definition within discharge letters.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.