gms | German Medical Science

Non-disclosive federated analysis conducts statistical analysis without sharing patient data between sites. As a tool for distributed computing, we use DataSHIELD [1] and extend its functionality towards the evaluation of prediction models on distributed data [2]. The objective is to calculate the Area Under the ROC curve (AUC), its confidence intervals, and to assess model calibration on distributed data. This presentation focuses on the technical implementation of the methodological details that were already presented elsewhere [3]. This work is part of the ProVal-MS study (https://difuture.de/, DRKS: 00014034) on patients with multiple sclerosis (MS) and aims to assess a newly developed treatment decision score with respect to the effect of an individual treatment on the patient’s disease progression. The patients were treated at five different sites.

The pooled analysis of distributed data commonly requires pooling data over sites, patient consent, and compliance with various data protection laws. To overcome data protection issues we developed R packages to enable distributed ROC analysis within the DataSHIELD framework. The analysis uses aggregated or noisy instead of sensitive individual data.

The package ds.predict.base encodes arbitrary R objects, send them from one to a second server. There they are decoded and loaded into the server’s R environment. This way, sharing of not just simple R objects (vectors) but also of data sets and models is possible. The package also enables calculation of prediction values or scores based on models and data locally available.

The second package ds.calibration builds on the first package and allows (https://github.com/difuture-lmu/ds.calibration) to assess calibration of pediction models. It implements the Brier score as well as plots of calibration curves of the respective scores. Here, we make use of the aggregation mechanism provided by DataSHIELD. The received aggregated values can be processed further locally to get global results.

Finally, ds.roc.glm (https://github.com/difuture-lmu/ds.roc.glm) conduts a distributed ROC analysis. It uses the ROC-GLM algorithm introduced by Pepe [4]. The main challenges of the distributed ROC-GLM algorithm is to calculate a global survivor function from individual scores and a distributed probit regression. Solutions are provided by aggregation techniques as well as methods from differential privacy [5].

We especially reevaluated the role of the DataSHIELD administrator. He is registering the assign and aggregate methods at the central server, defining what is allowed to get shared by these functions. If not set carefully, sensitive data could be shared. It is of utmost importance to assign an experienced scientist to that role. Nevertheless, the usage of DataSHIELD to enable distributed analysis is a milestone in data-driven medicine and can unlock a plethora of medical information for research.

Acknowledgement: DIFUTURE is funded as a part of the MI-I by the Bundesministerium für Bildung und Forschung (BMBF): 01ZZ1804C.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.