gms | German Medical Science

Kongress Medizin und Gesellschaft 2007

17. bis 21.09.2007, Augsburg

Assessing the prognostic accuracy of updated risk scores in a survival context

Meeting Abstract

Search Medline for

  • Rotraut Schoop - Universitätsklinikum Freiburg, Freiburg
  • Erika Graf - Universitätsklinikum Freiburg, Freiburg
  • Martin Schumacher - Universitätsklinikum Freiburg, Freiburg

Kongress Medizin und Gesellschaft 2007. Augsburg, 17.-21.09.2007. Düsseldorf: German Medical Science GMS Publishing House; 2007. Doc07gmds366

The electronic version of this article is the complete one and can be found online at:

Published: September 6, 2007

© 2007 Schoop et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.



Background: In many medical contexts, risk scores are used to describe patient risk profiles. Practicioners assume that updating these scores improves their prognostic accuracy. However, there exists as yet no universally acknowledged method to assess this assumed gain. Methods found in the literature range from inappropriate ad-hoc measures to ROC measures adapted from the diagnostic context to a survival setting. These, however, measure the discriminatory ability of the scores rather than their prognostic accuracy.

Material and Methods: We propose to measure the prognostic accuracy of an updated risk score in the following way: we consecutively estimate time-dependent "predictive values" P(T <= t + s | T > t, Z(t)) (with T being the time to event variable, t the time point of prediction, t+s the prediction horizon and Z(t) the value of the risk score as measured at time t), letting t and s vary. Estimation has to take place with an adequate model, e.g. a Cox model. The prognostic accuracy of the risk score for a specific time point of prediction t and prediction horizon t+s is then summarized by calculating the Brier score, i.e. the conditional mean squared error of prediction, of the predictive values described above.

We apply the methodology to data from a prospective cohort study on intensive care units where the endpoint of primary interest was death in intensive care. The study was conducted between February 2000 and July 2001 at the Charité University Hospital in Berlin and included 1768 patients with daily monitoring of patient data.

Results and Discussion: An improvement of predictive accuracy resulting from using updated information can be shown. This method is then contrasted to the ROC methodology mentioned above. Special emphasis is put on the different interpretability of the assessment measures.


Gerds TA, Schumacher M. Consistent estimation of the expected brier score in general survival models with right-censored event times. Biometrical J 2006; 48: 1029-1040.
Graf E, Schmoor C, Sauerbrei W, Schumacher M. Assessment and comparison of prognostic classification schemes for survival data. Stat in Med 1999; 18: 2229-2245.
Zheng Y, Heagerty, PJ. Semiparametric estimation of time-dependent ROC curves for longitudinal marker data. Biostat 2004; 5: 615-632.