gms | German Medical Science

Analysing medical studies variable selection methods are often applied in order to select one final model, which includes only prognostic factors or covariates with an influence on the outcome. However, often several models fit the data equally well, but might lead to different predictions for individual patients. Focusing on the selected model only this model selection uncertainty is ignored. Furthermore, parameter estimates obtained in the selected model may be biased and the length of confidence intervals can be underestimated substantially.

We account for model selection uncertainty by averaging over a set of possible models using weights estimated from bootstrap resampling [Ref. 3], [Ref. 1]. Results are compared to those obtained in the full model, by common backward selection strategies and bootstrap averaging [Ref. 2]. For illustration we use an example from the literature [e.g. [Ref. 4]] on the prediction of the percentage of body fat. In the framework of the linear regression model the different approaches are compared in a simulation study. We consider a model with 10 covariates (7 of them with an influence on the outcome), but will also present results from investigations with a larger number of noise variables (e.g. 25 covariates, 18 noise variables). It is shown that both, common variable strategies and bootstrap based averaging methods, lead to a better point prediction of the outcome variable than the full model, if the full model contains a lot of noise. However, wheras the length of confidence intervals is underestimated substantially with common variable selection strategies we obtained correct estimates of confidence intervals with bootstrap averaging methods.