gms | German Medical Science

Introduction: It is well known that shifted (differently distributed) effect measure modifiers (EMMs) are a potential cause of unequal average treatment effects in different populations and might therefore be a threat to the external validity of a study when one aims to infer to a specific target population not coinciding with the study sample [1]. Such a problem is framed in the literature as selection bias, a generalizability issue or a transportability issue [1]. In any case, the selection of covarites is crucial to identify the effect. However, the recommendations for covariate selection in the literature are conflicting. For instance, one recommendation is to include as many EMMs as possible, while another is to only include shifted EMMs. These recommendations are not in line with the theoretical literature which focuses on the causal relationships to select covariates [2]. We aim to bridge the gap between practical recommendations and theoretical necessity by outlining types of EMMs to (not) include.

Methods: To characterize different types of EMMs and their relation to the selection variable S (indicating the sample membership of source or target sample), we use directed acyclic graphs (DAGs), the four prototypical types of EMMs (direct, indirect, by proxy, by common cause) from VanderWeele and Robins [3] and a fifth type we refer to as an EMM by common effect. More precisely, we inspect how sufficient covariate sets for an unbiased estimation in the target sample differ depending on the types of EMMs and the direction of their relation to the selection variable (so S→EMM and EMM→S). Further, we simulate data to underpin the theoretical results by using estimators developed specifically for external validity evaluations [4], [5].

Results: When EMM→S, it is necessary to include all four prototypical types of EMMs in a sufficient set when they are shifted. However, when S→EMM, conditioning on shifted EMMs by proxy or common cause induces collider bias because the selection variable then takes the shape of an EMM by common effect. Hence, only EMMs that potentially interact with the treatment/exposure on the outcome should be included in the analysis when they are shifted. Non-shifted EMMs can be included in the analysis for EMM→S and S→EMM when they potentially interact with the treatment/exposure on the outcome. In contrast, non-shifted EMMs without the potential for interaction might induce collider bias due to an M-bias type structure. It follows that the sufficiency of a covariate set might depend on the direction of the selection variable and the types of EMMs. When a covariate set is only sufficient for EMM→S or S→EMM, we refer to this set as s-dependent.

Conclusion: Following current recommendations on covariate selection might bias an external validity analysis. For instance, including as many EMMs as possible or all shifted EMMs biases an estimation when parts of the covariate set are EMMs by proxy or common cause. Thus, researchers should focus on the causal relationships of the EMMs before conducting an external validity analysis, possibly by following the principles of covariate selection presented in this work.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.