gms | German Medical Science

66. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS), 12. Jahreskongress der Technologie- und Methodenplattform für die vernetzte medizinische Forschung e. V. (TMF)

26. - 30.09.2021, online

Findings from a Panel Discussion on Evaluation Methods in Medical Informatics

Meeting Abstract

  • Felix Holl - Hochschule Neu-Ulm, Neu-Ulm, Germany; Ludwig Maximilian Universität München, Munich, Germany; University of California, San Francisco, United States
  • Marina Fotteler - Hochschule für angewandte Wissenschaften Neu-Ulm, Neu-Ulm, Germany; Universität Ulm, Ulm, Germany
  • Stefan Müller-Mielitz - IEKF GmbH, Ibbenbüren, Germany
  • Walter Swoboda - Hochschule für angewandte Wissenschaften Neu-Ulm, Neu-Ulm, Germany

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. 66. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS), 12. Jahreskongress der Technologie- und Methodenplattform für die vernetzte medizinische Forschung e.V. (TMF). sine loco [digital], 26.-30.09.2021. Düsseldorf: German Medical Science GMS Publishing House; 2021. DocAbstr. 47

doi: 10.3205/21gmds043, urn:nbn:de:0183-21gmds0439

Published: September 24, 2021

© 2021 Holl et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at



Introduction: Healthcare systems globally are challenged by increasing costs caused by improved life expectancy and an increase in multimorbidity [1], [2]. The use of digital technology in healthcare delivery can help to combat this trend [3], [4].

There a substantial body of literature for the evaluation of medical informatics projects ranging from theoretical textbooks to reviews [5], [6], [7], [8]. The STARE-HI Statement on reporting of evaluation studies in Health Informatics published in 2008 provides a recommendation for evaluation results reporting [9]. However, despite the existing methods and literature, new technologies and applications for health are often not evaluated systematically, and when evaluations are performed, they often not generalized and focus on individual aspects [10]. Scholars have argued for the importance of a holistic, standardized, and comprehensive eHealth evaluation [11], [12]. Recent developments, like the introduction of the Digital Care Act in Germany, have further increased the importance of evaluation in medical informatics [13]. A scientific panel was held to discuss shortcomings of current evaluation methods used and discuss recommendations for new methods.

Methods: The panel had been accepted at the MIE 2020 conference, initially scheduled for April 2020, panels at the conference were canceled due to the COVID-19 pandemic and the panel was held online as part of the EFMI webinar series in July 2020. The seventy-minute-long panel consisted of panel presentations and a moderated expert discussion among panelists and participants. The panel was recorded and transcribed. The contents of the transcript were analyzed using grounded theory [14]. A synthesis of the online panel and critical discussion of the panel and current literature will be given in the talk.

Results: The four panel presentations gave the participants an overview of evaluation methods currently used in different medical informatics domains and their shortcomings. The presenters also highlighted new evaluation methods such as a roadmap for economic analysis of eHealth projects and the German Digital Care Act methods.

There was consensus among the panelists and participants in the expert discussion that the current evaluation methods used for digital solutions in medicine are inadequate and neglect several different aspects. Multiple participants stated that RCTs might not be the perfect solution and a focus need to be set on new evaluation methods. Since the panel, new literature has been published discussing the study designs used in the digital care act [15]. These methods include Continuous Evaluation of Evolving Behavioral Intervention Technologies [16].

Discussion: Participants discussed the shortcomings of randomized controlled trials and methods that need to be included in eHealth evaluation and called for new evaluation methods. The examples and the discussion showed weaknesses of the currently used methods and underlined the need for a new, evaluation standard for medical informatics. These shortcomings are underlined by new literature that has been published since the panel was held.

Conclusions: In light of the increasing importance of evaluation in medical informatics, driven by new legislation such as the Digital Care Act in Germany, this presentation will provide a valuable basis for discussion at the meeting and in the scientific community.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.


Keehan SP, Cuckler GA, Sisko AM, Madison AJ, Smith SD, Stone DA, et al. National health expenditure projections, 2014-24: Spending growth faster than recent trends. Health Aff. 2015;34(8):1407-17.
Rodriguez Santana I, Aragón MJ, Rice N, Mason AR. Trends in and drivers of healthcare expenditure in the English NHS: a retrospective analysis. Health Econ Rev. 2020;10(1):1–11.
Kannry J, Smith J, Mohan V, Levy B, Finnell J, Lehmann CU. Policy Statement on Clinical Informatics Fellowships and the Future of Informatics-Driven Medicine. Appl Clin Inform. 2020 Oct 28;11(5):710–3. DOI: 10.1055/s-0040-1717117 External link
Stenner SP, Chakravarthy R, Johnson KB, Miller WL, Olson J, Wickizer M, et al. Prescribing: Reducing costs through in-class therapeutic interchange. Appl Clin Inform. 2016 Dec 18;7(4):1168–81. DOI: 10.4338/ACI-2016-09-RA-0160 External link
Brender J. Handbook of Evaluation Methods for Health Informatics. Elsevier; 2006.
Eslami Andargoli A, Scheepers H, Rajendran D, Sohal A. Health information systems evaluation frameworks: A systematic review. Int J Med Inform. 2017;97:195–209. DOI: 10.1016/j.ijmedinf.2016.10.008 External link
Fanta GB, Pretorius L, Erasmus L. An evaluation of ehealth systems implementation frameworks for sustainability in resource constrained environments: A literature review. In: IAMOT 2015 - 24th Int Assoc Manag Technol Conf Technol Innov Manag Sustain Growth. 2015. p. 1046–63.
Friedman CP, Wyatt J. Evaluation methods in biomedical informatics. New York, NY: Springer New York; 2006. (Health Informatics). DOI: 10.1007/0-387-30677-3 External link
Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI-Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform. 2009;78(1):1–9.
Holl F, Swoboda W. Methods to Measure the Impact of mHealth Applications: Preliminary Results of a Scoping Review. Stud Health Technol Inform. 2018;251:285–8.
Catwell L, Sheikh A. Evaluating eHealth interventions: The need for continuous systemic evaluation. PLoS Med. 2009;6(8):1–6.
Enam A, Torres-Bonilla J, Eriksson H. Evidence-based evaluation of ehealth interventions: Systematic literature review. J Med Internet Res. 2018;20(11).
Gerke S, Stern AD, Minssen T. Germany’s digital health reforms in the COVID-19 era: lessons and opportunities for other countries. npj Digit Med. 2020;3(1):1–6.
Birks M, Mills J. Grounded Theory A Pratical Guide. 2nd ed. SAGE Publications Ltd; 2015.
Gensorowsky D, Lampe D, Hasemann L, Düvel J, Greiner W. “Alternative study designs” for the evaluation of digital health applications – a real alternative? Z Evid Fortbild Qual Gesundhwes. 2021;161:33–41.
Mohr DC, Cheung K, Schueller SM, Brown CH, Duan N. Continuous evaluation of evolving behavioral intervention technologies. Am J Prev Med. 2013;45(4):517–23.