gms | German Medical Science

The expected benefit of well-maintained metadata repositories lies in facilitating data integration and improving quality, as required data elements do not have to be re-specified, but existing high-quality predefined elements can be reused [1]. To yield the expected benefit, the included metadata must have sufficient quality. The manual verification of metadata elements is time-consuming and biased. Our aim is to apply quality criteria to metadata in healthcare and medical research settings and to develop computable validations. In order to be easily adaptable to new metadata repository systems, the quality criteria should support the standardized ISO 11179-3 metamodel [2], [3] which can be mapped to other metadata standards used in clinical research [1].

We carried out a literature review and applied the identified criteria to the metadata in the medical context, checking whether the metrics could be calculated automatically. Metrics developed by Oucha and Duval were computable, but not evaluated for a healthcare and medical research context [4]. We analyzed these metrics for their applicability in medical contexts and found that most of them need to be heavily adapted in order to work with healthcare metadata. For example, their approach works with free text fields, of which the ISO 11179-3 standard only contains two. McMahon and Denax proposed a framework for assessing metadata quality in epidemiological and public health research settings, but it was developed to guide human experts in reviewing the metadata and does not contain computable metrics [5]. Additionally, new, domain-specific criteria can be derived from clinical data quality criteria [6]. These criteria test the metadata’s ability to ensure data quality, e.g., by counting the presence of validation information. Other, more understandability-focused criteria, can be created by using specifics of the ISO 11179-3 standard, e.g., the number of distinctive values used as keys in slot annotations.

While there have been attempts establishing metadata quality criteria, none of them fulfill all our requirements. That is why we selected and adapted some criteria from the sources mentioned above: completeness, conformance to expectations, consistency, findability, timeliness and provenance. The newly developed criteria include the number of slots and distinctive keys in a repository, the proportion of numerical elements without units of measure, the length of definition values and the share of string elements as well as the portion of them that does not provide validation rules. The quality criteria can be evaluated automatically so that human reviewers can focus on potential problems and other, non-calculable criteria (e.g. the accuracy of definitions).

We performed a first evaluation of the developed metrics and could identify potential quality problems in a set of research metadata. More advanced methods of analysis, e.g., semantic analysis of free-text fields, are conceivable but would require semantic annotations and ontology support [7], which are not provided by current ISO 11179-3 conformant implementations like the Samply.MDR [8]. Furthermore, metadata repositories that support user authentication and provenance information would be able to calculate reputation scores for each user based on the quality of their submitted metadata and encourage them to improve their contributions.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.