gms | German Medical Science

51. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (gmds)

10. - 14.09.2006, Leipzig

A Framework for the Web-based Multi-method Evaluation of a Web-based Information System on Hospitals

Meeting Abstract

Search Medline for

  • Christa Weßel - Institut für Medizinische Informatik, RWTH Aachen, Aachen
  • Frederic Weymann - Institut für Medizinische Informatik, RWTH Aachen, Aachen
  • Cord Spreckelsen - Institut für Medizinische Informatik, RWTH Aachen, Aachen

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e.V. (gmds). 51. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. Leipzig, 10.-14.09.2006. Düsseldorf, Köln: German Medical Science; 2006. Doc06gmds015

The electronic version of this article is the complete one and can be found online at:

Published: September 1, 2006

© 2006 Weßel et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.



Background: The Need for Transparency in Health Care

The request for a high quality of medical services and for an economically sensible approach leads to the call for comprehensive, ubiquitous available and up-to-date data on the health care services. The focus lies on hospitals, which represent the largest medical service in health care. Potential patients, health care providers, health economics and other stakeholders in health care are known to use the internet for gaining information on hospitals [1].

German hospitals are obligated by law to publish quality reports in the internet from 2005 on, which are offered on several internet portals as PDFs [2], [3], [4]. However, it is time consuming to gather information of several hospitals on a certain question, to compare them and to use this as a base for decision making. Furthermore accurate and up-to-date data is necessary [5]. This is aggravated by the nearly continuously introduction of new laws or adapted service structures.

Thus a flexible and easy-to-use system is needed to offer hospital data in the continuously changing setting of health care [5]. First of all, the user requires a concise, intuitive website, which presents a comparative overview of several hospitals at a glance. This is rarely possible on the existing websites (see for example [3], [4], [6], [7]). The information has to be ready to serve as a tool to make a contribution to the informed consent [8].

Objective: Information in the Internet

Based on these and further common requirements for web-based information systems [9] the CERES Project implemented a prototype [10]. The prototype addresses the user-group potential patient. The central, object-oriented database is based on a meta-model of "the German hospital". The user is able to access this database by different web-frontends, which are "Search", "Tables", "Texts" and "Map".

The most crucial precondition is the availability of published data on German hospitals. This is given with the publication of the hospital quality reports [3], [4]. During a first evaluation period the website will offer the user some published data of sixty hospitals in Berlin, e.g. data from the quality reports. This covers a certain range of hospital types in a defined geographical area. The user is introduced to the information system by a short overview. He starts with the search and selects, whether he gets the information via tables, short texts or on a map. He can change between the data presentations whenever he wants. To learn, whether our object-oriented approach meets the needs of the user the web-based evaluation is the next step.

Method: Design of the Web-based Evaluation

The study design has to consider privacy [11] and follows the so far identified standards of scientific research in the internet [8], [12]. The evaluation is based upon three building blocks:

1) The combination of quantitative and qualitative research methods, known as triangulation, improves the quality of the study and grants insights on user's needs and requests [13], [14]. The identification of the appropriate data builds the first step. 2) For both, quantitative and qualitative data the data collection is based on a web application 3) Interfaces are specified to facilitate the data transfer for the quantitative and the qualitative data analysis.

Result: Technical Realization

We have implemented a web-based infrastructure for the evaluation according to the triangulation approach: Logfiles record the number of user visits and actions. An XML based questionnaire acquires quantitative data on the user profiles and their experiences and further wishes and ideas. Users may enter a comprehensive, natural language feedback via free text forms. The texts build the basis of data for the qualitative research.

The data of the questionnaire and the texts are stored in a MySQL database. The quantitative data are transferred to tools for the statistical analysis, for example SPSS. The texts representing the qualitative data are automatically exported to computer tools for qualitative text analysis, (here: MAXqda).

The evaluation results are monitored continuously. A first summative evaluation takes place after three months.

Conclusion and Outlook

By the evaluation of the described web-based information system on hospitals we want to follow the requirements outlined by [15]. We expect insights about how far we meet the users' needs on up-to-date, location-independent and easily accessible data, which is suitable for supporting their own decision making – and necessary improvements.

Further research activities will investigate the permanent improvement of the database schema, for example in order to meet the requirements of health economists or hospital managers. Within the scope of a research project a complete database of the contents of the quality reports of all German hospitals cannot be maintained constantly. Therefore we hope to form a project consortium of different stakeholders in the field in the nearer future in order to grant a sustainable development and maintenance of such an information system. This refers for example to the maintenance of the database and the platform and the guarantee of a 24/7 service.


The web-based information system on hospitals, its design and realization was and is only possible through the dedication of several students, graduates and post-graduates in computer science and medicine. We thank them for their cooperation, teamwork and proceeded interest in the project.


Hospital, World Wide Web, Information System, Decision Making, Informed Consent, Evaluation Studies, Quantitative Evaluation, Qualitative Research.


Eysenbach G. Consumer Health Informatics. BMJ. 2000; 230: 1713-16.
Sozialgesetzbuch Fünftes Buch. Gesetzliche Krankenversicherung - In der Fassung des Gesetzes zur Sicherung der nachhaltigen Finanzierungsgrundlagen der gesetzlichen Rentenversicherung Nachhaltigkeitsgesetz vom 21. Juli 2004 (BGBl. I S. 1791) §137 Qualitätssicherung bei zugelassenen Krankenhäusern. – last access Mar 8, 2006.
VdAK, AEV. Klinik-Lotse. – last access Mar 8, 2006.
VdAK. Qualitätsberichte Krankenhäuser. – last access Mar 8, 2006.
Klauber J, Robra BP, Schellschmidt H (Hg.). Krankenhaus-Report 2004. Schwerpunkt: Qualitätstransparenz - Instrumente und Konsequenzen. Stuttgart: Schattauer; 2005.
guideguide. Krankenhausguide. – last access Mar 8, 2006.
Initiativkreis Ruhrgebiet. Klinikführer Rhein-Ruhr 2005 2006. – last access Mar 8, 2006.
Eysenbach G, Wyatt J. Using the Internet for Surveys and Health Research. J Med Internet Res. 2002;4(2):e13. <URL:> last access. Mar 8, 2006.
Murugesan S, Desphande Y (Eds.). Web Engineering: Software Engineering and Web Application Development. Lecture Notes in Computer Science. Volume 2016 / 2001. Berlin, Heidelberg: Springer, 2001.
Weßel C, Ißler L, Spreckelsen C, Karakas G. Bürgerorientierte Präsentation von Informationen über Krankenhäuser im Internet mittels automatisierter Textgenerierung. In: Klar R, Köpcke W, Kuhn K, Lax H, Weiland S, Zaiß A (Eds.) gmds Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. 50. Jahrestagung, 12. - 15. September 2005. Freiburg: gmds dae; 2005. p. 117-119.
Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, Stalnaker N, Taplin SH, Sisk C, Ernster VL, Wilkie HA, Yankaskas B, Poplack SP, Urban N, West MM, Rosenberg RD, Michael S, Mercurio TD , Ballard-Barbash R. Current Medicolegal and Confidentiality Issues in Large, Multicenter Research Programs. American Journal of Epidemiology. 2000;152(4):371-378.
Eysenbach G. Improving the Quality of Web Surveys: The Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004;6(3):e34. <URL:> last access Mar 8, 2006.
Kaplan B. Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf. 2001; 64: 39-56.
Petter SC, Gallivan MJ. Toward a Framework for Classifying and Guiding Mixed Method Research in Information Systems. In: Proceedings of the 37th Hawaii International Conference on System Sciences. Los Alamitos: IEEE; 2004.
Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information system - reflections and lessons based an the HIS-EVAL workshop in Innsbruck. Int J Med Inf. 2004; 73 (6): 479-91.