gms | German Medical Science

Background: Hospitals are compared and categorized according to quality indicators including patient experiences with hospital care. This provider profiling is often linked to payment incentives and used by individual hospitals for quality improvement. However, measurement of patient experiences and hence the provider profile is influenced by many factors. For example, the level of random variation (due to varying sample sizes) and confounding variables (patient and hospital characteristics) play an important role. Therefore, one aim of provider profiling methods should be to ‘split the noise from the signal’, i.e. to identify the true variability between hospitals. Because of the significant impact of provider profiling on policy and practice, an effective, valid and reliable methodology is necessary.

Research Questions: Using an empirical sample of patients’ experiences collected through questionnaires we will examine whether commonly used patient experience quality indicators are appropriate for provider profiling purposes. Specifically, we will 1) calculate the two types of intra-class correlations to assess between-provider variability (ICC1) and reliability (ICC2) and 2) evaluate performance with different calculated (ICCs, 95%-CI, permutation test) and plotted (ICC1, distribution, 95%-CI) parameters.

Methods: For the application of provider profiling, we used patient survey data from a cross-sectional multi-center study of 23 Swiss hospitals with 123 units. The patient experience questionnaire included nine questions regarding the hospital stay using the HCAHPS survey and the Generic Short Patient Experiences Questionnaire. Using mixed effects models, we calculated and assessed two types of ICCs: the ICC1 to describe variability between units and hospitals and ICC2 to examine the reliability of the profiling. Additionally, to further scrutinize between-provider variance, we evaluated the permutation test of each ICC1 and the 95%-CIs of the repeatability estimates. Furthermore, we plotted each calculated ICC1 to quantify the level of variance and assess the distribution and shape of the results.

Results: Depending on the item, 1716 – 1863 (mean 1831) patients rated their experiences with a hospital stay. The ICC1 at the unit level ranged from 0.013 to 0.059 (mean 0.031) and from 0.009 to 0.035 (mean 0.023) at the hospital level indicating no to small between-provider variability at both levels. Plots for all nine items across the two levels found that two-third of the 95%-CIs of ICC1s were wide and included 0. The calculated ICC1 were further assessed with the plotted parameters, and only one third of the plots showed a normal distribution (a bell-shape and no skewness). At the unit level, ICC2 ranged from 0.62 to 0.885 (mean 0.691) and at the hospital level from 0.176 to 0.454 (mean 0.345) indicating moderate to good reliability at the unit level, and weak reliability at the hospital level.

Discussion: Our example questions common indicators of patient experience for provider profiling. The analyses point out that adequate ICCs for quality indicators are a prerequisite to get robust results for provider ranking. Even if the calculated ICC1 values were within accepted values (≥0.05) indicating variances between providers, the corresponding plots showed only few deviants of providers. Because plots sometimes provide a slightly different picture compared to calculated ICC1, all nine quality indicators also required the visual assessment in addition to the calculated ICCs. None of the indicators were suitable for provider profiling purposes. To get an accurate picture of differences between providers all methods need to be used simultaneously. Therefore, we recommend that ICC1 should not be used alone, but together with plots, evaluation of 95%-CIs and the distribution of repeatability estimates. In contrast, for most indicators ICC2 values were sufficient (≥0.8) at the unit level, but not at the hospital level. The results draw attention to the importance of sufficient sample size to ensure measurement of high reliability and a low degree of measurement errors to produce robust profiling.

Conclusion: Provider profiling based on patients’ experiences is part of how healthcare reimbursement is allocated e.g. in the U.S. Furthermore, based on published provider results, patients can choose which hospital they want to be admitted to. Therefore, the ability to assess variance among providers is crucial to identify positive or negative deviants, hence we need reliable measurements. Without sufficient ICCs, the use of patient experiences as quality indicators cannot be recommended.