gms | German Medical Science

Kongress Medizin und Gesellschaft 2007

17. bis 21.09.2007, Augsburg

Development and validation of a measurement instrument for appraising indicator quality: appraisal of indicators through research and evaluation (AIRE) instrument

Meeting Abstract

Suche in Medline nach

  • Johan de Koning - National Institute for Public Health and the Environment (RIVM), Bilthoven

Kongress Medizin und Gesellschaft 2007. Augsburg, 17.-21.09.2007. Düsseldorf: German Medical Science GMS Publishing House; 2007. Doc07gmds798

Die elektronische Version dieses Artikels ist vollständig und ist verfügbar unter:

Veröffentlicht: 6. September 2007

© 2007 de Koning.
Dieser Artikel ist ein Open Access-Artikel und steht unter den Creative Commons Lizenzbedingungen ( Er darf vervielfältigt, verbreitet und öffentlich zugänglich gemacht werden, vorausgesetzt dass Autor und Quelle genannt werden.



Background: In recent years, performance measurement in health care has become top priority for all major stakeholders in health care systems worldwide and large numbers of quality indicators have emerged from many sources. However, despite the availability of hundreds of indicators, the quality of many of them often is questioned.

Aim: The aim of this study was to develop and validate an instrument for assessing the quality of indicators.

Method: The AIRE Instrument was developed and tested through a multi-staged process including; literature review, item generation, selection and grouping, consultation rounds with experts in the field of performance measurement in health care, investigation of reliability and validity by an expert panel (n=8) using a set of clinical indicators, and a questionnaire survey among these experts on the instrument’s relevance and usability.

Results: The AIRE Instrument consists of 20 quality criteria (items) grouped into four quality domains: 1. Purpose, relevance and organizational context, 2. Stakeholder involvement, 3. Scientific evidence, 4. Additional evidence, formulation and usage. All panellists found the instrument useful for appraising indicator quality. Reliability was acceptable for most domains (Cronbach’s alpha 0.69–0.94). Intraclass correlations (ICC) were calculated to assess the reliability within each domain. All domains were rated consistently, however for different numbers of appraisers. As some items could be interpreted differently, particularly in domain 1 and domain 2, further refinement of these items was needed. Quality indicators produced as part of an established indicator program or organizations specialized in this field had significantly higher scores on domain 1 (p=,016), domain 3 (p=,046), and domain 4 (p=,001).

Conclusion: The AIRE Instrument can be used consistently to appraise indicator quality and is sensitive to differences in important aspects of clinical indicators. pplication of this instrument by individuals or institutions developing quality indicators enhances consistency in indicator development procedures and indicator quality.