gms | German Medical Science

Background: In many medical contexts, risk scores are used to describe patient risk profiles. Practicioners assume that updating these scores improves their prognostic accuracy. However, there exists as yet no universally acknowledged method to assess this assumed gain. Methods found in the literature range from inappropriate ad-hoc measures to ROC measures adapted from the diagnostic context to a survival setting. These, however, measure the discriminatory ability of the scores rather than their prognostic accuracy.

Material and Methods: We propose to measure the prognostic accuracy of an updated risk score in the following way: we consecutively estimate time-dependent "predictive values" P(T <= t + s | T > t, Z(t)) (with T being the time to event variable, t the time point of prediction, t+s the prediction horizon and Z(t) the value of the risk score as measured at time t), letting t and s vary. Estimation has to take place with an adequate model, e.g. a Cox model. The prognostic accuracy of the risk score for a specific time point of prediction t and prediction horizon t+s is then summarized by calculating the Brier score, i.e. the conditional mean squared error of prediction, of the predictive values described above.

We apply the methodology to data from a prospective cohort study on intensive care units where the endpoint of primary interest was death in intensive care. The study was conducted between February 2000 and July 2001 at the Charité University Hospital in Berlin and included 1768 patients with daily monitoring of patient data.

Results and Discussion: An improvement of predictive accuracy resulting from using updated information can be shown. This method is then contrasted to the ROC methodology mentioned above. Special emphasis is put on the different interpretability of the assessment measures.