gms | German Medical Science

GMS Medizinische Informatik, Biometrie und Epidemiologie

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e.V. (GMDS)

ISSN 1860-9171

Improving health IT monitoring via an electronic system for the exchange between science and practice

Verbesserung des IT-Monitorings im Gesundheitswesen durch ein elektronisches System zum Austausch zwischen Wissenschaft und Praxis

Research Article

  • Jan-Patrick Weiß - Department of Business Management and Social Sciences, Health Informatics Research Group, University of Applied Sciences Osnabrück, Germany
  • corresponding author Ursula Hübner - Department of Business Management and Social Sciences, Health Informatics Research Group, University of Applied Sciences Osnabrück, Germany
  • Jens Rauch - Department of Business Management and Social Sciences, Health Informatics Research Group, University of Applied Sciences Osnabrück, Germany
  • Johannes Thye - Department of Business Management and Social Sciences, Health Informatics Research Group, University of Applied Sciences Osnabrück, Germany
  • Frank Teuteberg - School of Business Administration and Economics, Faculty of Corporate Accounting and Information Systems, University Osnabrück, Germany
  • Jan Van der Linde - Department of Engineering and Computer Science, University of Applied Sciences Osnabrück, Germany
  • Frank Thiesing - Department of Engineering and Computer Science, University of Applied Sciences Osnabrück, Germany
  • Jan-David Liebe - Department of Business Management and Social Sciences, Health Informatics Research Group, University of Applied Sciences Osnabrück, Germany; Institute of Medical Informatics, University for Health Sciences, Medical Informatics and Technology (UMIT), Hall in Tyrol, Austria

GMS Med Inform Biom Epidemiol 2021;17(3):Doc11

doi: 10.3205/mibe000225, urn:nbn:de:0183-mibe0002253

Published: July 12, 2021

© 2021 Weiß et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at


Health information technology (IT) is a decisive factor for hospitals in optimizing the provision of healthcare services and many countries are interested in understanding, monitoring, and benchmarking the mechanisms of digital transformation on the national level. Part of this effort is the use of science-based maturity models to measure the level of digitization of healthcare institutions. The results of the maturity measurement are finally disseminated through science-practice dialogue concepts. The aim of this study is to develop and evaluate a technical platform to support this science-practice dialogue in the context of nationwide health IT monitoring in hospitals. Here users, the Chief Information Officer (CIO) in hospitals, provide data on the status of health IT in their institutions through a structured self-assessment on the platform. The data is analyzed and processed with scientific instruments, i.e. (1) according to scientifically established procedures of quantitative methodology and (2) in the context of current scientific findings and research questions.

The individual findings are presented to the user on a dashboard. The platform comprises the six components data capturing, indicators and scores, algorithms, storage, data-compiling pipeline and dashboard. All of them were developed pursuing the design science methodology. The evaluation of the platform revealed that it supports CIOs in IT strategic management. Usability tests showed that the platform increases user satisfaction. Within this study we could show how a science-practice dialogue to generate practical implications in the context of a nationwide health IT monitoring can be technically implemented.

Keywords: health IT monitoring, benchmarking, health information technology, information system, knowledge translation


Die Gesundheits-Informationstechnologie (IT) ist für Krankenhäuser ein entscheidender Faktor bei der Optimierung der Bereitstellung von Gesundheitsdienstleistungen. Viele Länder sind daran interessiert, die Mechanismen der digitalen Transformation auf nationaler Ebene zu verstehen, zu überwachen und zu benchmarken. Teil dieser Bemühungen ist die Nutzung von wissenschaftlich fundierten Reifegradmodellen, um den Digitalisierungsgrad der Gesundheitseinrichtungen zu messen. Die Ergebnisse der Reifegradmessung werden schließlich durch Konzepte des Wissenschaft-Praxis-Dialogs in die Breite getragen. Ziel dieser Studie sind die Entwicklung und Evaluation einer technischen Plattform zur Unterstützung dieses Wissenschaft-Praxis-Dialogs im Rahmen eines bundesweiten Gesundheits-IT-Monitorings in Krankenhäusern. Dabei stellen die Nutzer, die Chief Information Officer (CIO) in den Krankenhäusern, Daten zum Status der Gesundheits-IT in ihren Einrichtungen durch eine strukturierte Selbsteinschätzung auf der Plattform zur Verfügung. Die Daten werden mit wissenschaftlichen Instrumenten, das heißt (1) nach wissenschaftlich etablierten Verfahren der quantitativen Methodik und (2) im Kontext aktueller wissenschaftlicher Erkenntnisse und Fragestellungen, analysiert und verrechnet und den Anwendern in einem Dashboard individualisiert präsentiert.

Die Plattform besteht aus den sechs Komponenten Datenerfassung, Indikatoren und Scores, Algorithmen, Speicherung, Datenzusammenstellungspipeline und Dashboard. Sie wurde nach den Prinzipien von Design Science entwickelt. Eine Evaluation der Plattform zeigt, dass sie Chief Information Officer bei der strategischen Steuerung ihrer IT unterstützt. Usability-Tests weisen darauf hin, dass die Plattform die Zufriedenheit bei den Nutzern steigert. In dieser Studie konnten wir zeigen, wie ein Wissenschaft-Praxis-Dialog zur Generierung praktischer Implikationen im Rahmen eines landesweiten Gesundheits-IT-Monitorings technisch umgesetzt werden kann.

Schlüsselwörter: Gesundheits-IT-Monitoring, Benchmarking, Gesundheitsinformationstechnologie, Informationssysteme, Wissenstransfer


Health IT is a decisive factor for hospitals in optimizing the provision of healthcare services regarding effectiveness, quality and costs and is thus part of the discourse of strategic management [1]. Therefore, many countries are interested in understanding, monitoring, and benchmarking the mechanisms of digital transformation on the national level to face the increasing pressures for quality and cost in health care. Prominent examples are the Health Information Technology for Economic and Clinical Health (HITECH) Act of the USA [2], the Digital Maturity Assessment (DMA) of England [3] and the collaboration of the countries Denmark, Finland, Iceland, Norway, Sweden, the Faroe Islands, Greenland, and Åland in the Nordic eHealth benchmarking [4]. With the pass of the Hospital Future Act in September 2020, hospitals in Germany will receive a total of up to EUR 4.3 billion in funding by the government for modernization [5]. The funding program will be accompanied by the determination of the digital maturity level of the hospitals, which will be used to measure the transparency and comparability of the digitization progress.

The HITECH Act can also be seen as an example of how important the system for measuring digitization can be for a funding program, because here the model initially underestimated IT adoption rates and accordingly in the end of the program overestimated the impact of the program [6]. Furthermore, it is important that the collected data are used and presented optimally regarding the science-practice dialogue. The theoretical concept of the science-practice dialogue can be put into practice in three different ways. One approach for the science-practice dialogue is knowledge push, in which the pursuit of knowledge itself drives the scientific process, without the necessity to directly apply this new knowledge to solve problems within this process [7]. Knowledge-pull strives to solve a practical problem. However, this approach does not necessarily result in sustainable knowledge implementation due to the different cultural background of the two realms of science and practice [7]. The preferable science-practice dialogue to repeatedly monitoring health IT is the third approach which combines the models of knowledge-pull and push into a co-production of knowledge [8]. Information can be co-created by transforming data to information, i.e., pooling data from practice and processing them according to generally accepted rules and with scientific instruments. This co-production cycle is finally highlighted presenting the results to the practitioners so that they can incorporate the information into their body of knowledge and can act accordingly. If their new experience and knowledge is again shared with the scientists, the loop is closed. In the context of health IT, practice is mainly represented by the CIOs (and other IT decision makers) in hospitals, who want to use the data for the purpose of information management and decision support in their own organization [9] and by the political decision makers, who need the data for governing digitalisation in healthcare on a macrolevel. Science is represented by researchers who are interested to examine the antecedents and consequences of IT on the health care system with quantitative and longitudinal studies.

While there are different concepts to encourage the science-practice dialogue [8], [10], [11], [12], [13], [14], practical cooperation often lacks an appropriate exchange system that combines valid and reliable instruments with a technical platform to allow for information storage, processing and output in a secure environment. Whereas such systems do exist for clinical research using patient data [15], [16], only a few proprietary platforms for monitoring health IT are available e.g. HIMSS platform [17], Meisterworks platform [18], however, without aiming to improve research in understanding health IT. It is therefore the overall goal of this study to go beyond a mere technical platform and to provide and evaluate a combined science-practice exchange system that supports monitoring, modelling, and predicting health IT adoption, use, and maturity. Its task is to streamline the process from data gathering to information presentation and decision making based on knowledge, thereby shortening the data-to-knowledge cycle in which the knowledge is applied in practice and the resulting new data is in turn collected and used to generate new knowledge [19]. The technical solution chosen must be open to any kind of analysis including the longitudinal perspective using advanced statistical methods.

The overall aim of the study is broken down into the following research questions (RQ):

  • RQ 1. How can an exchange system be designed and technically realised to support the science-practice dialogue for monitoring health IT?
  • RQ 2. How do practitioners – here, CIOs of hospitals – evaluate the potentially added values of the exchange system in particular its dashboard interface?

Based on these findings this study intends to propose a model for a science-practice dialogue cycle.



The exchange system is meant to support the communication between science and practice for performing repeated scientific benchmarks of healthcare organisations and their digital performance. The requirements [20], [21] for such exchange systems resulted from an incremental evolutionary process that started with a benchmark carried out in 2011 (Figure 1 [Fig. 1]) to coordinate the information demands of IT managers of German hospitals [22]. The benchmark was repeated in 2013 and modified according to the feedback gained in 2011, which gave rise to the development of a hierarchically structured scoring system for measuring the degree of digitisation of clinical processes, i.e. the Workflow Composite Score System (WCS) [23], [24]. Finally, the evaluation of the 2013 system led to the specification of the exchange system of 2018 and its finalisation in 2019 including six components (C). The following requirements can be summarized based on the experience of the two previous iterations in 2011 and 2013 and in the preparations for the 2018 benchmark:

  • The calculation of the scores for each participant must be automated.
  • The individual score of each user must be shown together with his/her reference classes in a single diagram.
  • The dashboard user must be able to select between processes and reference groups on every score level.
  • The dashboard user must be able to select all indicators assigned to a single process-descriptor pair in a user-friendly manner from one screen.
  • The system must be modularised, so it is independent from the survey tools in use.
  • The collected data must be persistent and independent of the statistical analysis to avoid causalities or directional relationships.
  • The platform must allow for historisation (resp. longitudinal analyses) and consolidation of the data.
  • All software, frameworks and libraries must be open source.

Figure 1 [Fig. 1] shows the evolution of the components from a primarily manual procedure with only selected electronic tools (2011, 2013) to the targeted digitally integrated system of 2018 with the new components and their integration marked in grey. All these components should be embedded in a technical platform to provide a coherent environment for establishing and maintaining the communication through this system. In this sense, the exchange system should be more than a technical space but should exploit scientific knowledge and methods for practical use (Table 1 [Tab. 1]). RQ 1 addresses the interaction of all the components via the exchange system. Therefore, this paper focuses on the development of a technical platform to orchestrate these components and thereby makes use of previous work particularly regarding the components 1, 2 and 3 as well as component 4. The technical platform itself must realise the components 5 and 6 and should provide mechanisms for interfacing all components. Research question 2 specifically targets the dashboard, i.e., component 6, and its evaluation by CIOs and IT management related experts. Research question 2 also addresses the potential added value gained when using the exchange system.

Data from surveys

The existing data of the IT Report Health Care surveys from 2011 and 2013 [25], [26] were uploaded together with new data captured by the electronic questionnaire [27] which made use of the validated scores [23] (Table 2 [Tab. 2]). The integration of data from 2011 and 2013 should demonstrate the feasibility of the historicisation and consolidation of various research items. The new data stem from two studies. The first study surveyed chief information officers and IT management experts of German hospitals. The link to the electronic questionnaire tool was sent to 1,224 persons responsible for 1,950 institutions of which 224 hospitals (represented by the CIOs and the experts) took part and 197 participated in the Health IT Benchmarking 2018. In addition, a second study surveying clinical directors was conducted to measure the satisfaction of users and the perceived usefulness of the systems. The survey targeted medical and nursing directors of 1,951 German hospitals of which 355 hospitals (represented by the clinical directors) took part and 145 participated in the Health IT Benchmarking 2018. Furthermore, the data of quality reports from German hospitals for the years 2011 to 2016 and the demographic data of the German hospital register were pre-processed and uploaded. This procedure was meant to demonstrate the feasibility of consolidating primary data with secondary data from various sources.


In order to answer the second research question, a summative evaluation of the exchange system as such and its interactive dashboard interface was performed. The first aim was to obtain general feedback on the benchmarking itself and the scores used in the benchmark. This general feedback should include information about the extent to which the benchmarking results were used to support the communication between the IT managers and the board of executives. The second aim was to find out if the interaction with the dashboard was acceptable and the dashboard itself was usable. Usability was evaluated pursuant to the two-factor model of Zhang and Dran [28] based on Herzberg’s two-factor theory (hygiene factors and motivators) [29], [30], [31]. According to this theory, hygiene factors also known as “dissatisfiers”, are essential basic functions that are implicitly expected by the user, so that dissatisfaction arises if they are not fulfilled, but no satisfaction occurs if they are fulfilled [28], [30]. Motivators, on the other hand, have a positive influence on the user's perception if they are present, but their absence has no generally negative influence and, therefore, they are called “satisfiers” [28], [30]. An instrument was developed that operationalised eleven of the twelve categories adapting them to the dashboard interface. The instrument embraced 18 statements for hygiene factors and eight statements for motivator factors to be assessed on a five-point Likert scale (Table 3 [Tab. 3]). No statements were developed for the “enjoyment” category expressing “fun” and “humour” according to Zhang and Dran [28] because it was not deemed suitable for this application.


Exchange system

In order to answer research question 1 a technical platform was developed realising and integrating the six components to form the exchange system as a whole. The technical platform comprises four tiers (Figure 2 [Fig. 2]):

data source tier, i.e., integrating component 1,
data tier and
logic tier within the data warehouse system, i.e., implementing component 5, integrating component 4 and component 3 pursuant to component 2 and
presentation tier, i.e., implementing component 6.

The source systems were decoupled from the data warehouse system so that any data sources could be connected (Figure 3 [Fig. 3]). A data warehouse was set up to persist and consolidate the data. The data model combined the EAV model and the DV approach. It was thus independent of the analysis and, therefore, did not imply causalities or directional relationships for the variables that are based on the items from the survey data and used to calculate the scores for the benchmark. In the source layer, all data is transferred as raw data to the relational database of the data warehouse and archived for the future. The programming language R [32] was used mainly for advanced statistical analyses. The dashboard served as the interactive user interface of the exchange system. The electronic questionnaires were realised with LimeSurvey 2.54.3 [33] and Unipark [34]. PostgreSQL 9.6 [35] was used as database technology on Ubuntu Server 16.04 [36]. Extract, transform and load (ETL) processes were built with Pentaho Data Integration 7 [37] to load the data sources into the different layers of the data warehouse, transform the data to the data model and for partial calculations for the scores.

Additional processing of the data for the dashboard diagrams was realised through a connector written in Clojure [38]. These computations embraced e.g. calculating the various boundaries for the references groups is done and all this data is combined with the calculated scores from the data warehouse in a JavaScript Object Notation (JSON) format and made available under a RESTful (Representational state transfer) application programming interface (API) [39]. The RESTful API is used by the dashboard to request the calculated scores and other data e.g., indicator values, reference classes and demographic data, which are needed for the benchmarking diagrams, in a single hierarchical format via the two parameters User ID and year of participation (Figure 4 [Fig. 4]). The benchmarking diagrams were modelled with the library D3.js [40] and the TypeScript-based web framework Angular 6 [41] was used to reuse these new diagrams on different levels of the web interface. The access to the dashboard was secured by implementing the open source software Keycloak [42] for identity access management.

Figure 5 [Fig. 5], Figure 6 [Fig. 6] and Figure 7 [Fig. 7] show screens of the dashboard, which is divided into a navigation bar at the left and a canvas for the diagrams and additional information. When choosing the synopsis view, the Workflow Composite Score (WCS) as the main indicator for the degree of digital support for patient care processes is displayed (Figure 5 [Fig. 5]). The hospital's value is marked as a red bar within the distribution of all WCS values of one of the selected benchmarking reference groups (ownership or bed size). This distribution is divided into the innovation classes according to Rogers’ Diffusion of Innovation Theory [43], which are colour-coded in the diagram. The width of each of these classes indicates the absolute number of hospitals in this class. The next segment displays the individual process scores, which are visualised as a red line in a stacked bar chart with the two halves of the distribution marked with different grey shades. The type of reference group (ownership or size) can be selected in the blue box below the headline. In addition, IT satisfaction data from the clinical users of this organisation is displayed as a boxplot below.

More details can be retrieved when selecting one of the processes, e.g., admission (Figure 6 [Fig. 6]). The top diagram again shows the admission score depending on the reference group. The admission score is further broken down into the four descriptor sub-scores and is finally amended by a boxplot that depicts the user satisfaction of electronic information logistics for the admission process.

The descriptor sub-scores can be further detailed selecting a corresponding indicator and its diagram (Figure 7 [Fig. 7]). The top diagram again shows the selected descriptor sub-score depending on the reference group. For each indicator, user satisfaction with the functions that support the care process is displayed in a boxplot.


Out of all the hospitals that participated in the Health IT Benchmarking 2018, 35 IT managers, i.e., the practitioners, took part in the evaluation, which corresponded to a response rate of 17.8% (13.2% completed the full survey). Table 4 [Tab. 4] shows the percentage of IT managers who rated the comprehensibility of the diagrams and scores as “comprehensible” or “very comprehensible”.

Positive ratings in Table 4 [Tab. 4] exceeded the 80% limit for all types of diagrams. Answering the questions of how the benchmarking results were used (Table 5 [Tab. 5]) the participants reported that it served as an information source to be shared when communicating with colleagues (75.8%) and with the hospital management (66.7%) [n=33]. The benchmarking results were mostly used for the comparison of the status quo with other hospitals (85.7%), as a starting point for IT development (80%) and as an assessment of IT maturity (77.1%) [n=35]. When asked about new features, the participants reported that they would like to see specific experience reports (69.2%), get in contact with very well performing hospitals (57.7%) as well as obtain a link to national (61.5%) and international studies (42.3%) [n=26].

The following statements were made by the participants in the free text field, for example: “the currently approved project ‘Introduction of Digital Dictation’ was especially justified by the fact that we stood out (lagging behind) on this point in the study” or “sensitisation to the topic of ’digitisation’ in the corporate strategy. [...] A scientific and independent study with concrete comparisons of the competitors is a good opportunity to get ’attention’ and to bring the topic ’to mind’ to the top management once more.”. Furthermore, the majority of the participants (57.7%), wished to see their results within three months after submitting their data.

The median value for all hygiene factors was 4 (“agree”) and for the motivators 3.5 (between “neutral” and “agree”, Figure 8 [Fig. 8]). Both the hygiene factors and the motivators were rated high and above the neutral line of 3. The motivators were not rated similarly but a bit more towards the neutral line. Table 6 [Tab. 6] provides the Cronbach’s alpha values of both scales, which were well above 0.70 and thus spoke in favour of high internal consistency.

The hygiene factors’ median of 4 indicated that the basic needs for the web-based dashboard have largely been met but there was still room for improvements. The motivators achieved a median of 3.5, so that the current state of implementation significantly increased the user satisfaction with the web-based dashboard. Again, there was a need for further improvements.



The present study shows the technical integration and orchestration of the components of a technical platform, which is meant to enable an exchange system between science and practice to support CIOs in the strategic management of their IT. This exchange system was practically used for benchmarking the digital performance and the user satisfaction of 197 German hospitals. It thus provided feedback to the hospitals that had given IT related information via their CIOs and clinical directors in a first step and closed a first loop of science-practice interaction.

Addressing RQ 1 the components needed were transferred from a predominantly manual system into an integrated digital system by an iterative process. The data is exchanged via the questionnaire interface (C 1) and in turn aggregated information can be extracted via the dashboard (C 6). Both make use of a reliable and validated score system (C 2) so that data processing, i.e., the computation of the score system (C 3), could be fully automated. Comprehensive data management enables the consolidation and historicisation of different data sources (C 4). The different components were integrated with each other (C 5) so that the feedback cycle from collecting the data to providing the results (C 6) could be closed faster than before. The evaluation showed that most of the participants had expressed the wish to obtain feedback within three months after submitting their data. Therefore, compared to previous experience in the years 2011 and 2013, the technical platform leverages a faster process from data collection to information computation and visualisation.

Addressing RQ 2 the benchmarking information generated and presented by the exchange system was mainly used for the strategic communication and planning especially with the hospital’s top management as well as with IT staff and colleagues. Usability testing of the dashboard showed that hygiene factors and motivators were both indicating that there were no major issues dissatisfying the users and many indicators satisfying them. Hygiene factors and motivators were rated nearly similarly.

The study shows for the first time how a science-practice dialogue can be technically implemented within the framework of a nationwide health IT monitoring. It could be shown how the science-practice dialogue can generate scientifically founded strategic information for the management of hospital IT. A special focus was placed on the user-oriented visualization of the strategic information by means of a dynamic dashboard. This enabled extensive insights into the information needs of CIOs and IT-decision makers in healthcare organizations.


Although this platform has proved its use and usefulness it is still work in process towards a full exchange system. We will expand on this in the outlook. Furthermore, there are some technical improvements necessary that were revealed through the evaluation for example the integration of additional information like specific experience reports and national or international studies. Also, the automation processes of the platform are specific for the construction of the WCS system. If the WCS system changes, the automation tasks must also be adapted. Despite its capabilities to make use of longitudinal analyses such service has not been integrated into the dashboard. Another limitation of this study is the low response rate to the evaluation of the dashboard. This means that it can only be interpreted as a first indication. If, however, we combine these results with the previous findings based on the comments of 33 IT managers and CIOs [44], the picture is corroborated that substantial insights can be gained when using the exchange system. Even though hygiene factors and motivators were rated nearly similarly, further improvements of the visual design (access, availability, presentation and navigation) of the dashboard seem advisable.

Implications and outlook

Figure 9 [Fig. 9] shows the ideal exchange cycle with its different activities of science and practice when exchanging and accumulating data, information, and knowledge [11], [12], [13], [45]. Informed by [11] it signifies a new model for a science-practice-dialogue. This ideal cycle (Figure 9 [Fig. 9]), which goes beyond typical benchmarking, inspired the development of the exchange system, which in the current version, however, only partly realizes this cycle. The data to information exchange cycle is fully implemented while the transformation to knowledge, in particular the co-production of knowledge, needs further development.

CIOs use the information internally for their own purposes, i.e., mainly to make them aware of their status quo, for communication with others and decision making involving the top management. Communication problems between IT experts and executives can arise due to IT knowledge [46]. The exchange system can provide indicators and scores comprehensibly prepared for the management. This can help the CIOs to develop their line of argument when communicating with the top management. Hereby, diagrams can support verbal presentations and underpin the main message. As the free text fields in the evaluation showed the information extracted from the exchange system finally resulted in practical decisions, e.g., for the implementation of a dedicated application system, or in strategic decisions, e.g., to align the corporate strategy better with IT.

The conceptual taxonomy of knowledge transfer activities by Lomas [13] that distinguishes between “diffusion”, “dissemination” and “implementation” will guide the next steps of further developing the exchange system. Pursuant to this taxonomy the current stage of the exchange system would be best classified as “dissemination” where the research results are tailored to a particular audience [47]. While the evaluation results already proved that management changed its behaviour due to the information provided, this was no active process supported virtually by the technical platform but rather triggered by physical meetings between CIOs with other decision makers in their organization. Therefore, the next development steps will further steer the exchange system and more specifically the technical platform towards the category “implementation”, which is defined [13] by putting research results into practice in an active process. Allowing communities of practice [8] to share experience and knowledge in a collaborative approach together with scientists is part of this goal to foster the co-production of knowledge. For this purpose, existing knowledge, i.e., scientific publication and best practice reports, is to be displayed directly at the level of the key performance indicator.

In conclusion, this technical platform implements scientific research instruments and integrates authentic data from practice, so that the information provided is relevant both from a scientific and a practical point of view. The digitalisation of this science-practice dialogue through this exchange system ensures a more systematic and scalable process of collecting, processing and transforming data into new findings. This data-driven procedure is meant to bind science and practice in a steady and sustainable manner and thereby overcome the problems of exchanging and communicating between the two different realms of science and practice. Altogether, the study provides a comprehensive basis for the technical realisation of a national health IT monitoring platform.


Competing interests

The authors declare that they have no competing interests.


This study was funded by the Ministry of Science and Culture of the State of Lower Saxony, Germany (grant no. ZN 3062).


We would like to thank Moritz Esdar for conducting the IT Report Health Care survey and defining the computation of the WCS. We would also like to thank Michael Tissen for contributing to the developments of the web-based dashboard. We wish to thank all IT managers/CIOs for participating in this study.


Gholami R, Añón Higón D, Emrouznejad A. Hospital performance: Efficiency or quality? Can we have both with IT? Expert Syst Appl. 2015;42(12):5390-400. DOI: 10.1016/j.eswa.2014.12.019 External link
Adler-Milstein J, Jha AK. HITECH Act Drove Large Gains In Hospital Electronic Health Record Adoption. Health Aff (Millwood). 2017 Aug;36(8):1416-22. DOI: 10.1377/hlthaff.2016.1651 External link
Johnston DS. Digital maturity: are we ready to use technology in the NHS? Future Healthc J. 2017 Oct;4(3):189-92. DOI: 10.7861/futurehosp.4-3-189 External link
Nøhr C, Koch S, Vimarlund V, Gilstad H, Faxvaag A, Hardardottir GA, Andreassen HK, Kangas M, Reponen J, Bertelsen P, Villumsen S, Hyppönen H. Monitoring and Benchmarking eHealth in the Nordic Countries. Stud Health Technol Inform. 2018;247:86-90.
Kucera M. Krankenhauszukunftsgesetz: Milliarden für die Zukunft. kma – Klinik Management aktuell. 2020;25(10):34-5. DOI: 10.1055/s-0040-1718815 External link
Everson J, Rubin JC, Friedman CP. Reconsidering hospital EHR adoption at the dawn of HITECH: implications of the reported 9% adoption of a “basic” EHR. J Am Med Inform Assoc. 2020 Aug;27(8):1198-205. DOI: 10.1093/jamia/ocaa090 External link
Dilling L, Lemos MC. Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environ Chang. 2011;21(2):680-9. DOI: 10.1016/j.gloenvcha.2010.11.006 External link
Roux DJ, Rogers KH, Biggs HC, Ashton PJ, Sergeant A. Bridging the science-management divide: Moving from unidirectional knowledge transfer to knowledge interfacing and sharing. Ecol Soc. 2006;11(1):4. Available from: External link
Thye J, Esdar M, Liebe JD, Jahn F, Winter A, Hübner U. Professionalism of Information Management in Health Care: Development and Validation of the Construct and Its Measurement. Methods Inf Med. 2020 Jun;59(S01):e1-e12. DOI: 10.1055/s-0040-1712465 External link
Carlile PR, Rebentisch ES. Into the Black Box: The Knowledge Transformation Cycle. Manag Sci. 2003;49(9):1180-95. DOI: 10.1287/mnsc.49.9.1180.16564 External link
Liew A. Understanding data, information, knowledge and their inter-relationships. J Knowl Manag Pract. 2007;8(2):1-16.
Liyanage C, Elhag T, Ballal T, Li Q. Knowledge communication and translation – a knowledge transfer model. J Knowl Manag. 2009;13(3):118-31. DOI: 10.1108/13673270910962914 External link
Lomas J. Diffusion, dissemination, and implementation: who should do what? Ann N Y Acad Sci. 1993 Dec;703:226-35; discussion 235-7. DOI: 10.1111/j.1749-6632.1993.tb26351.x External link
Wehrens R. Beyond two communities – from research utilization and knowledge translation to co-production? Public Health. 2014 Jun;128(6):545-51. DOI: 10.1016/j.puhe.2014.02.004 External link
Athey BD, Braxenthaler M, Haas M, Guo Y. tranSMART: An Open Source and Community-Driven Informatics and Data Sharing Platform for Clinical and Translational Research. AMIA Jt Summits Transl Sci Proc. 2013 Mar 18;2013:6-8.
Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, Kohane I. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc. 2010 Mar-Apr;17(2):124-30. DOI: 10.1136/jamia.2009.000893 External link
HIMSS – Healthcare Information and Management Systems Society. EMRAM model: A strategic roadmap for effective EMR adoption and usage. 2017 [cited 2019 Jul 24]. Available from: External link
Moeller H. Meisterworks platform. [cited 2020 Nov 3]. Available from: External link
Friedman CP, Rubin JC, Sullivan KJ. Toward an Information Infrastructure for Global Health Improvement. Yearb Med Inform. 2017 Aug;26(1):16-23. DOI: 10.15265/IY-2017-004 External link
Weiß JP, Hübner U, Rauch J, Hüsers J, Teuteberg F, Esdar M, Liebe JD. Implementing a Data Management Platform for Longitudinal Health Research. Stud Health Technol Inform. 2017;243:85-9.
Rauch J, Weiss JP, Teuteberg F, Hübner U. Konsolidierte Datenmodellierung von Versorgungsdaten mit dem Entity-Attribute-Value-Modell und Data Vault [Consolidated data modeling of health services research data with the entity-attribute-value model and data vault]. GMS Med Inform Biom Epidemiol. 2017;13(1):Doc03. DOI: 10.3205/mibe000170 External link
Liebe JD, Egbert N, Frey A, Hübner U. Characteristics of german hospitals Adopting health IT systems – Results from an empirical study. In: Moen A, editor. User centred networked health care: Proceedings of MIE 2011 (23rd International Conference of the European Federation for Medical Informatics); 2011 Aug 28-31; Oslo, Norway. Amsterdam: IOS Press; 2011. (Studies in health technology and informatics; 169). p. 335-8.
Liebe JD, Hübner U, Straede MC, Thye J. Developing a Workflow Composite Score to Measure Clinical Information Logistics. A Top-down Approach. Methods Inf Med. 2015;54(5):424-33. DOI: 10.3414/ME14-02-0025 External link
Liebe JD, Hübner U. Developing and trialling an independent, scalable and repeatable IT-benchmarking procedure for healthcare organisations. Methods Inf Med. 2013;52(4):360-9. DOI: 10.3414/ME12-02-0016 External link
Hübner U, Liebe JD, Egbert N, Frey A. IT-Report Gesundheitswesen. Schwerpunkt IT im Krankenhaus. 2012 [cited 2019 Jan 22]. Available from: External link
Hübner U, Liebe JD, Straede MC, Thye J. IT-Report Gesundheitswesen. Schwerpunkt IT-Unterstützung klinischer Prozesse. 2014 [cited 2019 Jan 22]. Available from: External link
Hübner U, Liebe JD, Egbert N, Thye J, Esdar M, Hüsers J, Rauch J, Weiß JP. IT-Report Gesundheitswesen: Schwerpunkt: Wie reif ist die IT in deutschen Krankenhäusern? 2018 [cited 2019 Jan 22]. Available from: External link
Zhang P, Dran GM von. Satisfiers and dissatisfiers: A two-factor model for website design and evaluation. J Am Soc Inform Sci. 2000;51(14):1253-68. DOI: 10.1002/1097-4571(2000)9999:9999<:AID-ASI1039>3.0.CO;2-O External link
Herzberg FI. Work and the nature of man. Cleveland: World Publishing; 1966.
Herzberg F. One more time: How do you motivate employees? Harvard Bus Rev. 1968 Jan-Feb:53-62.
Herzberg F. One more time: How do you motivate employees? Harvard Bus Rev. 1987 Sep-Oct:109-20.
The R Foundation. The R Project for Statistical Computing. [cited 2021 Jan 23]. Available from: External link
LimeSurvey GmbH. LimeSurvey. [cited 2021 Jan 23]. Available from: External link
Tivian. UNIPARK. [cited 2021 Jan 23]. Available from: External link
The PostgreSQL Global Development Group. PostgreSQL. [cited 2021 Jan 23]. Available from: External link
Canonical Ltd. Ubuntu. [cited 2021 Jan 23]. Available from: External link
Hitachi Vantara Corporation. Pentaho Data Integration. [cited 2021 Jan 23]. Available from: External link
Hickey R. Clojure. [cited 2021 Jan 23]. Available from: External link
VMware. Spring Framework. [cited 2021 Jan 23]. Available from: External link
Bostock M. D3.js. [cited 2021 Jan 23]. Available from: External link
Google LLC. Angular. [cited 2021 Jan 23]. Available from: External link
Keycloak. [cited 2021 Jan 23]. Available from: External link
Rogers EM. Diffusion of Innovations. 5th ed. New York: Free Press; 2010.
Weiß JP, Thye J, Rauch J, Tissen M, Esdar M, Teuteberg F, Hübner UH. IT-Benchmarking als Zusammenspiel von Wissenschaft und Praxis – ein Web-Portal zur Dissemination individueller Ergebnisse für Krankenhäuser. In: Drews P, Funk B, Niemeyer P, Xie L, editors. Data driven X-Turning Data into Value. Multikonferenz der Wirtschaftsinformatik (MKWI 2018); 2018 Mar 6-9; Lüneburg, Germany. p. 659-70.
Gagnon ML. Moving knowledge to action through dissemination and exchange. J Clin Epidemiol. 2011 Jan;64(1):25-31. DOI: 10.1016/j.jclinepi.2009.08.013 External link
Thye J, Hübner U, Hüsers J, Babitsch B. IT Decision Making in German Hospitals - Do CEOs Open the Black Box? Stud Health Technol Inform. 2017;243:112-16.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13-24. DOI: 10.1002/chp.47 External link
MacKenzie SB, Podsakoff PM, Podsakoff NP. Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Q. 2011;35(2):293-334. DOI: 10.2307/23044045. External link
Bucher T, Dinter B. Process Orientation of Information Logistics An Empirical Analysis to Assess Benefits, Design Factors, and Realization Approaches. In: IEEE, editor. Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008); 2008 Jan 07-10; Waikoloa, Hawaii. DOI: 10.1109/HICSS.2008.361 External link
Delone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manag Inform Syst. 2003;19(4):9-30. DOI: 10.1080/07421222.2003.11045748 External link
Saaty TL. What is the analytic hierarchy process? In: Mitra G, Greenberg HJ, Lootsma FA, Rijkaert MJ, Zimmermann HJ, editors. Mathematical models for decision support. Berlin: Springer Verlag; 1988. p. 109-21.
Saaty TL. Decision making with the analytic hierarchy process. Int J Serv Sci. 2008;1(1):83-98. DOI: 10.1504/IJSSCI.2008.017590 External link
Löper D, Klettke M, Bruder I, Heuer A. Integrating healthcare-related information using the entity-attribute-value storage model. In: He J, Liu X, Krupinski EA, Xu G, editors. Health information science. Proceedings First International Conference HIS 2012; 2012 Apr 8-10; Beijing, China. Berlin, Heidelberg: Springer; 2012. (Lecture Notes in Computer Science; 7231). p. 13-24. DOI: 10.1007/978-3-642-29361-0_4 External link
Linstedt D, Olschimke M. Building a scalable data warehouse with Data Vault 2.0. Waltham, MA: Morgan Kaufmann; 2016.
Dresch A, Lacerda DP, Antunes Jr JAV. Design Science Research. Cham: Springer International Publishing; 2015. DOI: 10.1007/978-3-319-07374-3 External link