gms | German Medical Science

Health IT is a decisive factor for hospitals in optimizing the provision of healthcare services regarding effectiveness, quality and costs and is thus part of the discourse of strategic management [1]. Therefore, many countries are interested in understanding, monitoring, and benchmarking the mechanisms of digital transformation on the national level to face the increasing pressures for quality and cost in health care. Prominent examples are the Health Information Technology for Economic and Clinical Health (HITECH) Act of the USA [2], the Digital Maturity Assessment (DMA) of England [3] and the collaboration of the countries Denmark, Finland, Iceland, Norway, Sweden, the Faroe Islands, Greenland, and Åland in the Nordic eHealth benchmarking [4]. With the pass of the Hospital Future Act in September 2020, hospitals in Germany will receive a total of up to EUR 4.3 billion in funding by the government for modernization [5]. The funding program will be accompanied by the determination of the digital maturity level of the hospitals, which will be used to measure the transparency and comparability of the digitization progress.

The HITECH Act can also be seen as an example of how important the system for measuring digitization can be for a funding program, because here the model initially underestimated IT adoption rates and accordingly in the end of the program overestimated the impact of the program [6]. Furthermore, it is important that the collected data are used and presented optimally regarding the science-practice dialogue. The theoretical concept of the science-practice dialogue can be put into practice in three different ways. One approach for the science-practice dialogue is knowledge push, in which the pursuit of knowledge itself drives the scientific process, without the necessity to directly apply this new knowledge to solve problems within this process [7]. Knowledge-pull strives to solve a practical problem. However, this approach does not necessarily result in sustainable knowledge implementation due to the different cultural background of the two realms of science and practice [7]. The preferable science-practice dialogue to repeatedly monitoring health IT is the third approach which combines the models of knowledge-pull and push into a co-production of knowledge [8]. Information can be co-created by transforming data to information, i.e., pooling data from practice and processing them according to generally accepted rules and with scientific instruments. This co-production cycle is finally highlighted presenting the results to the practitioners so that they can incorporate the information into their body of knowledge and can act accordingly. If their new experience and knowledge is again shared with the scientists, the loop is closed. In the context of health IT, practice is mainly represented by the CIOs (and other IT decision makers) in hospitals, who want to use the data for the purpose of information management and decision support in their own organization [9] and by the political decision makers, who need the data for governing digitalisation in healthcare on a macrolevel. Science is represented by researchers who are interested to examine the antecedents and consequences of IT on the health care system with quantitative and longitudinal studies.

While there are different concepts to encourage the science-practice dialogue [8], [10], [11], [12], [13], [14], practical cooperation often lacks an appropriate exchange system that combines valid and reliable instruments with a technical platform to allow for information storage, processing and output in a secure environment. Whereas such systems do exist for clinical research using patient data [15], [16], only a few proprietary platforms for monitoring health IT are available e.g. HIMSS platform [17], Meisterworks platform [18], however, without aiming to improve research in understanding health IT. It is therefore the overall goal of this study to go beyond a mere technical platform and to provide and evaluate a combined science-practice exchange system that supports monitoring, modelling, and predicting health IT adoption, use, and maturity. Its task is to streamline the process from data gathering to information presentation and decision making based on knowledge, thereby shortening the data-to-knowledge cycle in which the knowledge is applied in practice and the resulting new data is in turn collected and used to generate new knowledge [19]. The technical solution chosen must be open to any kind of analysis including the longitudinal perspective using advanced statistical methods.

The overall aim of the study is broken down into the following research questions (RQ):

• RQ 1. How can an exchange system be designed and technically realised to support the science-practice dialogue for monitoring health IT?
• RQ 2. How do practitioners – here, CIOs of hospitals – evaluate the potentially added values of the exchange system in particular its dashboard interface?

Based on these findings this study intends to propose a model for a science-practice dialogue cycle.

The exchange system is meant to support the communication between science and practice for performing repeated scientific benchmarks of healthcare organisations and their digital performance. The requirements [20], [21] for such exchange systems resulted from an incremental evolutionary process that started with a benchmark carried out in 2011 (Figure 1 [Fig. 1]) to coordinate the information demands of IT managers of German hospitals [22]. The benchmark was repeated in 2013 and modified according to the feedback gained in 2011, which gave rise to the development of a hierarchically structured scoring system for measuring the degree of digitisation of clinical processes, i.e. the Workflow Composite Score System (WCS) [23], [24]. Finally, the evaluation of the 2013 system led to the specification of the exchange system of 2018 and its finalisation in 2019 including six components (C). The following requirements can be summarized based on the experience of the two previous iterations in 2011 and 2013 and in the preparations for the 2018 benchmark:

• The calculation of the scores for each participant must be automated.
• The individual score of each user must be shown together with his/her reference classes in a single diagram.
• The dashboard user must be able to select between processes and reference groups on every score level.
• The dashboard user must be able to select all indicators assigned to a single process-descriptor pair in a user-friendly manner from one screen.
• The system must be modularised, so it is independent from the survey tools in use.
• The collected data must be persistent and independent of the statistical analysis to avoid causalities or directional relationships.
• The platform must allow for historisation (resp. longitudinal analyses) and consolidation of the data.
• All software, frameworks and libraries must be open source.

Figure 1 [Fig. 1] shows the evolution of the components from a primarily manual procedure with only selected electronic tools (2011, 2013) to the targeted digitally integrated system of 2018 with the new components and their integration marked in grey. All these components should be embedded in a technical platform to provide a coherent environment for establishing and maintaining the communication through this system. In this sense, the exchange system should be more than a technical space but should exploit scientific knowledge and methods for practical use (Table 1 [Tab. 1]). RQ 1 addresses the interaction of all the components via the exchange system. Therefore, this paper focuses on the development of a technical platform to orchestrate these components and thereby makes use of previous work particularly regarding the components 1, 2 and 3 as well as component 4. The technical platform itself must realise the components 5 and 6 and should provide mechanisms for interfacing all components. Research question 2 specifically targets the dashboard, i.e., component 6, and its evaluation by CIOs and IT management related experts. Research question 2 also addresses the potential added value gained when using the exchange system.

The existing data of the IT Report Health Care surveys from 2011 and 2013 [25], [26] were uploaded together with new data captured by the electronic questionnaire [27] which made use of the validated scores [23] (Table 2 [Tab. 2]). The integration of data from 2011 and 2013 should demonstrate the feasibility of the historicisation and consolidation of various research items. The new data stem from two studies. The first study surveyed chief information officers and IT management experts of German hospitals. The link to the electronic questionnaire tool was sent to 1,224 persons responsible for 1,950 institutions of which 224 hospitals (represented by the CIOs and the experts) took part and 197 participated in the Health IT Benchmarking 2018. In addition, a second study surveying clinical directors was conducted to measure the satisfaction of users and the perceived usefulness of the systems. The survey targeted medical and nursing directors of 1,951 German hospitals of which 355 hospitals (represented by the clinical directors) took part and 145 participated in the Health IT Benchmarking 2018. Furthermore, the data of quality reports from German hospitals for the years 2011 to 2016 and the demographic data of the German hospital register were pre-processed and uploaded. This procedure was meant to demonstrate the feasibility of consolidating primary data with secondary data from various sources.

In order to answer the second research question, a summative evaluation of the exchange system as such and its interactive dashboard interface was performed. The first aim was to obtain general feedback on the benchmarking itself and the scores used in the benchmark. This general feedback should include information about the extent to which the benchmarking results were used to support the communication between the IT managers and the board of executives. The second aim was to find out if the interaction with the dashboard was acceptable and the dashboard itself was usable. Usability was evaluated pursuant to the two-factor model of Zhang and Dran [28] based on Herzberg’s two-factor theory (hygiene factors and motivators) [29], [30], [31]. According to this theory, hygiene factors also known as “dissatisfiers”, are essential basic functions that are implicitly expected by the user, so that dissatisfaction arises if they are not fulfilled, but no satisfaction occurs if they are fulfilled [28], [30]. Motivators, on the other hand, have a positive influence on the user's perception if they are present, but their absence has no generally negative influence and, therefore, they are called “satisfiers” [28], [30]. An instrument was developed that operationalised eleven of the twelve categories adapting them to the dashboard interface. The instrument embraced 18 statements for hygiene factors and eight statements for motivator factors to be assessed on a five-point Likert scale (Table 3 [Tab. 3]). No statements were developed for the “enjoyment” category expressing “fun” and “humour” according to Zhang and Dran [28] because it was not deemed suitable for this application.

In order to answer research question 1 a technical platform was developed realising and integrating the six components to form the exchange system as a whole. The technical platform comprises four tiers (Figure 2 [Fig. 2]):

1.

data source tier, i.e., integrating component 1,

2.

data tier and

3.

logic tier within the data warehouse system, i.e., implementing component 5, integrating component 4 and component 3 pursuant to component 2 and

4.

presentation tier, i.e., implementing component 6.

The source systems were decoupled from the data warehouse system so that any data sources could be connected (Figure 3 [Fig. 3]). A data warehouse was set up to persist and consolidate the data. The data model combined the EAV model and the DV approach. It was thus independent of the analysis and, therefore, did not imply causalities or directional relationships for the variables that are based on the items from the survey data and used to calculate the scores for the benchmark. In the source layer, all data is transferred as raw data to the relational database of the data warehouse and archived for the future. The programming language R [32] was used mainly for advanced statistical analyses. The dashboard served as the interactive user interface of the exchange system. The electronic questionnaires were realised with LimeSurvey 2.54.3 [33] and Unipark [34]. PostgreSQL 9.6 [35] was used as database technology on Ubuntu Server 16.04 [36]. Extract, transform and load (ETL) processes were built with Pentaho Data Integration 7 [37] to load the data sources into the different layers of the data warehouse, transform the data to the data model and for partial calculations for the scores.

Additional processing of the data for the dashboard diagrams was realised through a connector written in Clojure [38]. These computations embraced e.g. calculating the various boundaries for the references groups is done and all this data is combined with the calculated scores from the data warehouse in a JavaScript Object Notation (JSON) format and made available under a RESTful (Representational state transfer) application programming interface (API) [39]. The RESTful API is used by the dashboard to request the calculated scores and other data e.g., indicator values, reference classes and demographic data, which are needed for the benchmarking diagrams, in a single hierarchical format via the two parameters User ID and year of participation (Figure 4 [Fig. 4]). The benchmarking diagrams were modelled with the library D3.js [40] and the TypeScript-based web framework Angular 6 [41] was used to reuse these new diagrams on different levels of the web interface. The access to the dashboard was secured by implementing the open source software Keycloak [42] for identity access management.

Figure 5 [Fig. 5], Figure 6 [Fig. 6] and Figure 7 [Fig. 7] show screens of the dashboard, which is divided into a navigation bar at the left and a canvas for the diagrams and additional information. When choosing the synopsis view, the Workflow Composite Score (WCS) as the main indicator for the degree of digital support for patient care processes is displayed (Figure 5 [Fig. 5]). The hospital's value is marked as a red bar within the distribution of all WCS values of one of the selected benchmarking reference groups (ownership or bed size). This distribution is divided into the innovation classes according to Rogers’ Diffusion of Innovation Theory [43], which are colour-coded in the diagram. The width of each of these classes indicates the absolute number of hospitals in this class. The next segment displays the individual process scores, which are visualised as a red line in a stacked bar chart with the two halves of the distribution marked with different grey shades. The type of reference group (ownership or size) can be selected in the blue box below the headline. In addition, IT satisfaction data from the clinical users of this organisation is displayed as a boxplot below.

More details can be retrieved when selecting one of the processes, e.g., admission (Figure 6 [Fig. 6]). The top diagram again shows the admission score depending on the reference group. The admission score is further broken down into the four descriptor sub-scores and is finally amended by a boxplot that depicts the user satisfaction of electronic information logistics for the admission process.

The descriptor sub-scores can be further detailed selecting a corresponding indicator and its diagram (Figure 7 [Fig. 7]). The top diagram again shows the selected descriptor sub-score depending on the reference group. For each indicator, user satisfaction with the functions that support the care process is displayed in a boxplot.

Out of all the hospitals that participated in the Health IT Benchmarking 2018, 35 IT managers, i.e., the practitioners, took part in the evaluation, which corresponded to a response rate of 17.8% (13.2% completed the full survey). Table 4 [Tab. 4] shows the percentage of IT managers who rated the comprehensibility of the diagrams and scores as “comprehensible” or “very comprehensible”.

Positive ratings in Table 4 [Tab. 4] exceeded the 80% limit for all types of diagrams. Answering the questions of how the benchmarking results were used (Table 5 [Tab. 5]) the participants reported that it served as an information source to be shared when communicating with colleagues (75.8%) and with the hospital management (66.7%) [n=33]. The benchmarking results were mostly used for the comparison of the status quo with other hospitals (85.7%), as a starting point for IT development (80%) and as an assessment of IT maturity (77.1%) [n=35]. When asked about new features, the participants reported that they would like to see specific experience reports (69.2%), get in contact with very well performing hospitals (57.7%) as well as obtain a link to national (61.5%) and international studies (42.3%) [n=26].

The following statements were made by the participants in the free text field, for example: “the currently approved project ‘Introduction of Digital Dictation’ was especially justified by the fact that we stood out (lagging behind) on this point in the study” or “sensitisation to the topic of ’digitisation’ in the corporate strategy. [...] A scientific and independent study with concrete comparisons of the competitors is a good opportunity to get ’attention’ and to bring the topic ’to mind’ to the top management once more.”. Furthermore, the majority of the participants (57.7%), wished to see their results within three months after submitting their data.

The median value for all hygiene factors was 4 (“agree”) and for the motivators 3.5 (between “neutral” and “agree”, Figure 8 [Fig. 8]). Both the hygiene factors and the motivators were rated high and above the neutral line of 3. The motivators were not rated similarly but a bit more towards the neutral line. Table 6 [Tab. 6] provides the Cronbach’s alpha values of both scales, which were well above 0.70 and thus spoke in favour of high internal consistency.

The hygiene factors’ median of 4 indicated that the basic needs for the web-based dashboard have largely been met but there was still room for improvements. The motivators achieved a median of 3.5, so that the current state of implementation significantly increased the user satisfaction with the web-based dashboard. Again, there was a need for further improvements.

The present study shows the technical integration and orchestration of the components of a technical platform, which is meant to enable an exchange system between science and practice to support CIOs in the strategic management of their IT. This exchange system was practically used for benchmarking the digital performance and the user satisfaction of 197 German hospitals. It thus provided feedback to the hospitals that had given IT related information via their CIOs and clinical directors in a first step and closed a first loop of science-practice interaction.

Addressing RQ 1 the components needed were transferred from a predominantly manual system into an integrated digital system by an iterative process. The data is exchanged via the questionnaire interface (C 1) and in turn aggregated information can be extracted via the dashboard (C 6). Both make use of a reliable and validated score system (C 2) so that data processing, i.e., the computation of the score system (C 3), could be fully automated. Comprehensive data management enables the consolidation and historicisation of different data sources (C 4). The different components were integrated with each other (C 5) so that the feedback cycle from collecting the data to providing the results (C 6) could be closed faster than before. The evaluation showed that most of the participants had expressed the wish to obtain feedback within three months after submitting their data. Therefore, compared to previous experience in the years 2011 and 2013, the technical platform leverages a faster process from data collection to information computation and visualisation.

Addressing RQ 2 the benchmarking information generated and presented by the exchange system was mainly used for the strategic communication and planning especially with the hospital’s top management as well as with IT staff and colleagues. Usability testing of the dashboard showed that hygiene factors and motivators were both indicating that there were no major issues dissatisfying the users and many indicators satisfying them. Hygiene factors and motivators were rated nearly similarly.

The study shows for the first time how a science-practice dialogue can be technically implemented within the framework of a nationwide health IT monitoring. It could be shown how the science-practice dialogue can generate scientifically founded strategic information for the management of hospital IT. A special focus was placed on the user-oriented visualization of the strategic information by means of a dynamic dashboard. This enabled extensive insights into the information needs of CIOs and IT-decision makers in healthcare organizations.

Although this platform has proved its use and usefulness it is still work in process towards a full exchange system. We will expand on this in the outlook. Furthermore, there are some technical improvements necessary that were revealed through the evaluation for example the integration of additional information like specific experience reports and national or international studies. Also, the automation processes of the platform are specific for the construction of the WCS system. If the WCS system changes, the automation tasks must also be adapted. Despite its capabilities to make use of longitudinal analyses such service has not been integrated into the dashboard. Another limitation of this study is the low response rate to the evaluation of the dashboard. This means that it can only be interpreted as a first indication. If, however, we combine these results with the previous findings based on the comments of 33 IT managers and CIOs [44], the picture is corroborated that substantial insights can be gained when using the exchange system. Even though hygiene factors and motivators were rated nearly similarly, further improvements of the visual design (access, availability, presentation and navigation) of the dashboard seem advisable.

Figure 9 [Fig. 9] shows the ideal exchange cycle with its different activities of science and practice when exchanging and accumulating data, information, and knowledge [11], [12], [13], [45]. Informed by [11] it signifies a new model for a science-practice-dialogue. This ideal cycle (Figure 9 [Fig. 9]), which goes beyond typical benchmarking, inspired the development of the exchange system, which in the current version, however, only partly realizes this cycle. The data to information exchange cycle is fully implemented while the transformation to knowledge, in particular the co-production of knowledge, needs further development.

CIOs use the information internally for their own purposes, i.e., mainly to make them aware of their status quo, for communication with others and decision making involving the top management. Communication problems between IT experts and executives can arise due to IT knowledge [46]. The exchange system can provide indicators and scores comprehensibly prepared for the management. This can help the CIOs to develop their line of argument when communicating with the top management. Hereby, diagrams can support verbal presentations and underpin the main message. As the free text fields in the evaluation showed the information extracted from the exchange system finally resulted in practical decisions, e.g., for the implementation of a dedicated application system, or in strategic decisions, e.g., to align the corporate strategy better with IT.

The conceptual taxonomy of knowledge transfer activities by Lomas [13] that distinguishes between “diffusion”, “dissemination” and “implementation” will guide the next steps of further developing the exchange system. Pursuant to this taxonomy the current stage of the exchange system would be best classified as “dissemination” where the research results are tailored to a particular audience [47]. While the evaluation results already proved that management changed its behaviour due to the information provided, this was no active process supported virtually by the technical platform but rather triggered by physical meetings between CIOs with other decision makers in their organization. Therefore, the next development steps will further steer the exchange system and more specifically the technical platform towards the category “implementation”, which is defined [13] by putting research results into practice in an active process. Allowing communities of practice [8] to share experience and knowledge in a collaborative approach together with scientists is part of this goal to foster the co-production of knowledge. For this purpose, existing knowledge, i.e., scientific publication and best practice reports, is to be displayed directly at the level of the key performance indicator.

In conclusion, this technical platform implements scientific research instruments and integrates authentic data from practice, so that the information provided is relevant both from a scientific and a practical point of view. The digitalisation of this science-practice dialogue through this exchange system ensures a more systematic and scalable process of collecting, processing and transforming data into new findings. This data-driven procedure is meant to bind science and practice in a steady and sustainable manner and thereby overcome the problems of exchanging and communicating between the two different realms of science and practice. Altogether, the study provides a comprehensive basis for the technical realisation of a national health IT monitoring platform.

The authors declare that they have no competing interests.

This study was funded by the Ministry of Science and Culture of the State of Lower Saxony, Germany (grant no. ZN 3062).

We would like to thank Moritz Esdar for conducting the IT Report Health Care survey and defining the computation of the WCS. We would also like to thank Michael Tissen for contributing to the developments of the web-based dashboard. We wish to thank all IT managers/CIOs for participating in this study.