gms | German Medical Science

EbM in Qualitätsmanagement und operativer Medizin
8. Jahrestagung des Deutschen Netzwerks Evidenzbasierte Medizin e. V.

Deutsches Netzwerk Evidenzbasierte Medizin e. V.

22.03. - 24.03.2007 in Berlin

Adjustment lowers Agreement and Precision of Results of Medical Examination

Meeting Abstract

Search Medline for

EbM in Qualitätsmanagement und operativer Medizin. 8. Jahrestagung des Deutschen Netzwerks Evidenzbasierte Medizin e. V.. Berlin, 22.-24.03.2007. Düsseldorf, Köln: German Medical Science; 2007. Doc07ebm058

The electronic version of this article is the complete one and can be found online at: http://www.egms.de/en/meetings/ebm2007/07ebm058.shtml

Published: March 15, 2007

© 2007 Friedrich-Wedig et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.


Outline

Text

Hintergrund

Rankings are used as result of assessment. An example for rankings of classes are marks. Marks result from the share of solution of stadardized tests. Adjustment ist usual as calibration, if a threshold error has to be corrected. The adjustment of factors as number of “staff of the faculty” and “logarithm of number of inhibitants of the city” lowers the agreement of rankings. Adjustment is a method for focussing, for demonstration not for increase of accuracy.

Methoden

Data of Zimmermann 2006 [1]

Calculation of Bland 1986 [2]

Precision of Calculation

Up to 10 07 iterations in the procedure of calculation of the t-value.

Plots of results

Figure 1 [Fig. 1], Figure 2 [Fig. 2], Figure 3 [Fig. 3].

Ergebnisse

Zimmermann’s comparison of results of the 4th semester examination of medical students (2006) are here investigated with the method of Bland (1986). Orinal ranking and adjusted ranking differ between –17 and +11 (n= 36). The space of two standard deviations count between –10 and 39. So the agreement of the ranking lists is 18 – 10 = 8 up to 18 + 39 = 49. This in not enough for equal use of original ranking and adjusted ranking.

The 0.95 confidence interval is 18,5 +/- (2.03 * 10.2) = [-2, +39]. And this interval is wider than the 36point original ranking. Because of the right rotating tilting of linear regression the weaker universities profit from the adjustment and the success of the best universities seems to bei smaller.

The connection between the success in objective tests and disturbance of factors which are part of the adjusted ranking is in matter of scientific and technological questions low: y = 0,78 x + 0,40 with R2 = 0,61.

Schlussfolgerung/Implikation

You must not use adjustment as a method for comparison of results of medical examination. Few intruments are needed for detection of pitfalls of rankings. Blands method is easy, quick and smart.


References

1.
Zimmermann R, Wegscheider K, van den Bussche H. Medizinische Fakultäten. Der Ausbildungserfolg im Vergleich (I). Dtsch Ärztebl. 2006;103(25):A-1732-A-1738.
2.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;307-10.