gms | German Medical Science

Introduction: Healthcare systems globally are challenged by increasing costs caused by improved life expectancy and an increase in multimorbidity [1], [2]. The use of digital technology in healthcare delivery can help to combat this trend [3], [4].

There a substantial body of literature for the evaluation of medical informatics projects ranging from theoretical textbooks to reviews [5], [6], [7], [8]. The STARE-HI Statement on reporting of evaluation studies in Health Informatics published in 2008 provides a recommendation for evaluation results reporting [9]. However, despite the existing methods and literature, new technologies and applications for health are often not evaluated systematically, and when evaluations are performed, they often not generalized and focus on individual aspects [10]. Scholars have argued for the importance of a holistic, standardized, and comprehensive eHealth evaluation [11], [12]. Recent developments, like the introduction of the Digital Care Act in Germany, have further increased the importance of evaluation in medical informatics [13]. A scientific panel was held to discuss shortcomings of current evaluation methods used and discuss recommendations for new methods.

Methods: The panel had been accepted at the MIE 2020 conference, initially scheduled for April 2020, panels at the conference were canceled due to the COVID-19 pandemic and the panel was held online as part of the EFMI webinar series in July 2020. The seventy-minute-long panel consisted of panel presentations and a moderated expert discussion among panelists and participants. The panel was recorded and transcribed. The contents of the transcript were analyzed using grounded theory [14]. A synthesis of the online panel and critical discussion of the panel and current literature will be given in the talk.

Results: The four panel presentations gave the participants an overview of evaluation methods currently used in different medical informatics domains and their shortcomings. The presenters also highlighted new evaluation methods such as a roadmap for economic analysis of eHealth projects and the German Digital Care Act methods.

There was consensus among the panelists and participants in the expert discussion that the current evaluation methods used for digital solutions in medicine are inadequate and neglect several different aspects. Multiple participants stated that RCTs might not be the perfect solution and a focus need to be set on new evaluation methods. Since the panel, new literature has been published discussing the study designs used in the digital care act [15]. These methods include Continuous Evaluation of Evolving Behavioral Intervention Technologies [16].

Discussion: Participants discussed the shortcomings of randomized controlled trials and methods that need to be included in eHealth evaluation and called for new evaluation methods. The examples and the discussion showed weaknesses of the currently used methods and underlined the need for a new, evaluation standard for medical informatics. These shortcomings are underlined by new literature that has been published since the panel was held.

Conclusions: In light of the increasing importance of evaluation in medical informatics, driven by new legislation such as the Digital Care Act in Germany, this presentation will provide a valuable basis for discussion at the meeting and in the scientific community.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.