gms | German Medical Science

51. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (gmds)

10. - 14.09.2006, Leipzig

Is there sufficient evidence for evidence-based informatics?

Meeting Abstract

Search Medline for

  • Elske Ammenwerth - UMIT - Private Universität für Gesundheitswissenschaften, Med. Informatik und Technik Tirol, Hall in Tirol
  • Christof Machan - UMIT - Private Universität für Gesundheitswissenschaften, Med. Informatik und Technik Tirol, Hall in Tirol

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e.V. (gmds). 51. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. Leipzig, 10.-14.09.2006. Düsseldorf, Köln: German Medical Science; 2006. Doc06gmds049

The electronic version of this article is the complete one and can be found online at:

Published: September 1, 2006

© 2006 Ammenwerth et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.




Evidence-based health informatics is the conscientious, explicit and judicious use of current best evidence when making decisions about information technology in health care. The term has been introduced in the last years, and it has been argued that evidence-based health informatics is an important step towards better healthcare IT [1], [2]. This argumentation copies from the experiences with Evidence-Based Medicine (EBM) that was initiated in the 1990s to support health-care decisions by providing the best evidence available. It is hoped that methods of EBM such as meta-analysis can also be successfully applied in evidence-based health informatics. In fact, some publications successfully used the technique of meta-analysis to assess e.g. the impact of home-monitoring on systolic and diastolic blood pressure [3]. However, a meta-analysis of study results needs very homogeneous studies, evaluating well-defined interventions in comparable settings using standardized effect measures. There is reason to doubt this assumption in health informatics. The aim of this work is to discuss the usefulness of EBM methods in health informatics based on an investigation of CPOE evaluation studies.


Computerized physician order entry systems (CPOE) have been in the center of interest recently, as their potential to significantly improve patient care by reducing medication errors and ADEs (adverse drug events) has been described frequently [4]. We conducted a comprehensive literature search based on PubMed to identify all studies that evaluated CPOE systems in a clinical setting. We classified all studies according to their effect measures using the classification system previously published [5]. We then selected the three largest groups with comparable effect measures for a quantitative meta-analysis. Finally, we contrasted the result of the meta-analysis with other available qualitative and uncontrolled studies, and we analyzed reviews on CPOE.


We were able to identify around 200 CPOE evaluation studies. From the quantitative and controlled studies, 12 investigated the effect of CPOE on prescription costs, 21 analysed the effect of CPOE on medication errors, and 9 studies the effect of CPOE on ADEs. For those three groups, we conducted a quantitative meta-analysis. In each of the three groups the preliminary results of the meta-analysis point to a clear positive effect of CPOE (i.e. CPOE can reduce prescription costs, medication errors and ADEs). For example, only one ADE study showed partly negative effects of CPOE on ADE. In contrast, the analysis of uncontrolled and qualitative evaluation group showed scientific evidence of negative effects of CPOE on medication errors and ADE rates (e.g. [6], [7]). Some reviews point to the fact that evidence on CPOE effects is often based on non-commercial, home-grown systems [8], and that positive effects of CPOE depend on various context factors such as leadership and project management [9].


A quantitative meta-analysis based on quantitative controlled studies pointed to largely positive effects of CPOE on outcome quality and costs. A contrasting analysis of other studies (uncontrolled case studies, qualitative studies) showed evidence of negative effects not reflected in the quantitative controlled studies. There may be several reasons for this difference of published quantitative controlled evidence and other research results. A lot of time and energy is invested to introduce CPOE, hoping to improve health care. No or negative outcome is likely not to be published (negative publication bias). Quantitative research is oriented versus hypotheses the researcher tries to verify. He may overlook unintended consequences on variables not included in the study protocol. CPOE introduction is a long process of optimization of the system, the workflow, the organization and the support. It would not be surprising when researchers presented results only after the expected outcome has been reached and the hypotheses confirmed (see a corresponding example analyzed in [10]). This would mean that CPOE systems are optimized as long as needed to achieve positive outcome. Authors may tend to report only on those variables where they found a positive effect, and drop those where no effect was visible for publication. Furthermore, the majority of evaluated CPOE systems are non-commercial systems, developed and operated by organizations that have direct access to the underlying software. This enables them to achieve maximum results with regard to software quality. Finally, CPOE systems tend to heavily interrupt traditional workflow, explaining a lot of resistance by clinicians. It is likely that researchers will only choose settings providing optimal preconditions for CPOE introduction: high motivation of the staff, low complexity of workflow, best information technology used. This is not representative for the “typical” setting. All this may explain why published quantitative evidence shows mostly positive effects. In fact, the effects of CPOE seem to be over-estimated, or, in other words, published quantitative evidence presents the maximum positive effects of CPOE (in perfect conditions), but not the range of possible (positive and negative) effects.


There is a discrepancy between published quantitative controlled evidence (mostly pointing to positive effects of CPOE) and other scientific evidence especially from qualitative research or non-controlled studies (often showing negative effects and problems with CPOE introduction). Some reasons have been discussed. It seems that published quantitative evidence highlights the potential of CPOE achievable in perfect conditions, but not the day-to-day reality. It seems to overestimate the positive effects and underestimate possible negative effects. That means that evidence-based informatics only relying on published quantitative controlled studies may overlook important CPOE effects. Overall, it seems necessary to include findings from both, quantitative controlled trials as well as from other scientific sources such as qualitative research in future meta-analysis. Adequate methods for this triangulative meta-analysis have to be developed.


Rigby M. Evaluation: 16 Powerful Reasons Why Not to Do It - And 6 Over-Riding Imperatives. In: Patel V, Rogers R, Haux R, editors. Proceedings of the 10th World Congress on Medical Informatics (Medinfo 2001). Amsterdam: IOS Press; 2001. p. 1198-202.
Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using A Comprehensive Health Technology Assessment Framework. Comput Biol Med. 2002;32(3):165-77.
Cappuccio FP, Kerry SM, Forbes L, Donald A. Blood pressure control by home monitoring: meta-analysis of randomised trials. Bmj. 2004;329(7458):145.
Kuperman GJ, Gibson RF. Computer physician order entry: benefits, costs, and issues. Ann Intern Med. 2003;139(1):31-9.
Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982 - 2002. Methods Inf Med. 2005;44:44-56.
Koppel R, Metlay J, Cohen A, Abaluck B, Localio A, SE K, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293(10):1197-2003.
Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF. High rates of adverse drug events in a highly computerized hospital. Arch Intern Med. 2005;165(10):1111-6.
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409-16.
Sittig DF, Stead WW. Computer-based physician order entry: the state of the art. J Am Med Inform Assoc. 1994;1(2):108-23.
Berger RG, Kichak JP. Computerized physician order entry: helpful or harmful? J Am Med Inform Assoc. 2004;11(2):100-3.