gms | German Medical Science

Background: Outcomes of individual consultants in 10 specialties specialty were published over the summer of 2013/14 in response to the NHS Commissioning Board’s “Everyone Counts” publication.

This includes data Adult Cardiac, Vascular, Thyroid and Endocrine, Bariatric, Orthopaedic, Urological, Bowel, Oesophago-Gastric and Head and Neck surgery as well as Interventional Cardiology. The release of such data, meant to inform patient choice, by NHS England is unprecedented in scope and highly controversial.

Research question: We examine whether the data recently published is likely to improve patient choice, and discuss unintended consequences of the publicly released data.

Methods: We conducted a documentary/content analysis of all publicly available web reports of the specialties participating in the Consultant Outcomes Publication in the NHS, complemented with structured interviews amongst participating specialties, to complement and verify data and observations.

We examined participating characteristics (specialties, procedures, and consultants included, and consultants opting out), metric properties of the reports (suitability of outcome measures. baseline prevalence of outcome, number of outliers identified), and mode of presentation of results (navigability, graphical presentation of statistical results).

Results: The NHS Consultant Outcome Publication showed wide variability in the type and scope of data presented and in the approach to formal presentation. The release of outcome data demonstrates differences in coverage, use of measures unsuitable for reporting (e.g. mortality rates <1% after hip replacement), and major differences in the navigability of the web reports.

Perhaps unintended, the publication augmented existing methodological issues in performance benchmarking, namely implausible effects of risk adjustment where case volume is low, uncontrolled selection bias, and type-II errors potentially leading to false complacency: amongst thousands of surgeons only 6 statistical outliers were identified (leaving patients to assume that all others surgeons perform well).

Discussion: The publication of individual clinician’s outcomes has been controversial and at current the data is unsuitable to support patient choice. In particular, the data is not aligned with patients’ expectations to learn about particular skills and knowledge of a consultant, new promising treatments, a differentiation of the best surgeon within a team/trust (vs an analytical approach that compares surgeon performance to a national expected mean).

Practical implications: The current initiative is not fit for purpose. Paradoxically, it may possibly lead to better data quality and, eventually, outcomes by bringing to light key methodological issues which are currently being formalized in revised guidance and research on public reporting and benchmarking.