gms | German Medical Science

Introduction: Surgical video data provide valuable insight into the course of the surgery. The application of machine learning algorithms enables diverse analyzation of the content such as surgical phase, gestures, instrument, and anatomical structures recognition [1]. This is utilized in the Surgomics concept where videos of minimally invasive surgery are processed to extract defined surgomic features that aim at predicting post-surgery complications of the patient together with further clinical data [2], [3]. Therefore, a data pipeline is required to provide the video data and along with clinical data for the patient.

Methods: The objective is to provide an infrastructure capable of promptly informing a surgeon about potential complications after the surgery is completed, while also ensuring the necessary data privacy for cross-institutional analyses. The concept of the pipeline was compiled in an iterative process considering various requirements including data security and the integration into existing infrastructures for linking the video and clinical data. The (pseudonymized) clinical data is provided by the MeDIC (Medical Data Integration Center) Heidelberg which also consumes the video data and extracted surgomic features.

For the realization of the data pipeline, Talend Data Integration is used as a main tool. Here, docker container and python scripts are imbedded to execute the machine learning models that are applied to the surgical videos.

Results: The pipeline has two alternative paths to process the captured videos. For one, a real-time capable platform is constructed to analyze videos on a medical PC connected to the operating room infrastructure where the video signal is captured directly. The conventional approach, with a focus on long-term storage, uses the infrastructure of the MeDIC Heidelberg. First, the post-surgery transmitted video is de-identified by blacking out frames that were recorded outside of the body. Then, the features are extracted where the same analysis as in the parallel approach is used. The results are stored as FHIR resources. The surgomic features are exported together with additional clinical patient data to a project-specific FHIR server. This server serves as the data repository for the project, where data can be stored and retrieved, for example for complication prediction, which is calculated based on the available data. Additionally, the server is the communication interface to mobile endpoints where the results are presented to clinical staff.

Discussion: The pipeline offers an automated solution for initiating the analysis of surgical videos, utilizing the camera signal directly in the operating room and also post-transmission of the recorded videos. Nevertheless, the two parallel processing paths result in redundant analysis. The medical PC within the operating room addresses the lack of real-time processing while also providing a location-independent solution. This setting was simulated with identical hardware outside the operating room where a push notification was sent within five minutes after finishing the procedure. The conventional, more sustainable path allows for long-term storage and re-use of the videos due to the applied de-identification.

Conclusion: The used approach presents a feasible way to automatically process surgical videos and enables the combination of the results with existing clinical data.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.