Article
Automation of multiple instance orchestration of modular web platforms for research
Search Medline for
Authors
Published: | August 19, 2022 |
---|
Outline
Text
Introduction: FAIRification2 [1] tools that are designed to represent the research output of large scientific consortia are still limited in functionality and are still being actively developed. Since 2012 we designed and implemented the menoci3 web portal. menoci addresses the needs of FAIR representation of biomedical basic research data and has been successfully implemented for several large consortia at Göttingen Campus [2], [3]. Operation of multiple instances in parallel requires efficient measures to avoid errors, e.g. in update processes, and to reduce administrative workload. Here, we describe our approach to achieve a maximum level of seamless automation in server operation and software updating procedures.
Methods: Servers for the different menoci instances are hosted on virtual machines (VM) using IT infrastructure of the local academic IT-service provider GWDG (https://www.gwdg.de). The menoci source code is hosted and maintained in the Göttingen Campus GitLab service. A basic Continuous Integration/Continuous Delivery (CI/CD) pipeline based on GitLab CI/CD is applied to routinely build and schedule updated Docker images from latest source code revisions, the upstream Drupal Docker image and other Drupal modules used in menoci. menoci Docker images are automatically pushed to the GitLab Container Registry by a CI/CD pipeline, where they are publicly available. The CI/CD pipeline is run by a GitLab runner within a Docker container for flexible deployment and scalability. GitLab functionality for code reviews is employed, using protected branches and eligible approvers for merge requests.
Results: Early menoci development was mainly driven by the implementation of additional modules, features and optimization of user experience to fulfill the researchers’ requirements. Since rollout of menoci to an increasing list of research consortia, we additionally focused on improving performance, software quality and enhanced automation processes. Our process combines updates for menoci modules, database components and the Drupal content management system as well as Drupal modules integrated in menoci functionalities. Furthermore, all code changes to menoci can automatically be deployed to all VMs in a Docker image with the CI/CD pipeline. Success or failure of update processes is monitored systematically to facilitate error handling. All processes are extensively documented to easily integrate new team members into administrative tasks.
Discussion: For our use cases, the development of enhanced automation processes has proven to be key to reduce workload for technical administrative tasks. However, certain challenges remain to be solved, such as: First, IT personnel resources dedicated to server operation for research are scarcely available in academia. Also optimization of basic server managing tasks is time consuming and benefits are often barely visible for the target user groups. Second, although the different menoci instances share a wide range of their components, each use case requires an individually adapted setup of modules as well as interfaces with additional digital infrastructures. This complicates the construction of automation pipelines as update processes diverge. Third, version leaps of major IT components such as the Drupal content management system require large additional workloads that are hardly compatible with available resources.
Funding: This project was funded partially by the DFG through projects KFO5002 (project 426671079), CRC 1002 infrastructure (INF) project, as well as the Z projects of CRC 1190 and CRC 1286, Volkswagenstiftung in the MTB-Report project (ZN3424) and Germany’s Excellence Strategy - EXC 2067/1- 390729940.
The authors declare that they have no competing interests.
The authors declare that an ethics committee vote is not required.
References
- 1.
- Wilkinson MD, Dumontier M, Mons B, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. 2016;3:160018.
- 2.
- Suhr M, Lehmann C, Bauer CR, Bender T, Knopp C, Freckmann L, et al. Menoci: lightweight extensible web portal enhancing data management for biomedical research projects. BMC Bioinformatics. 2020;21(1):582.
- 3.
- Kusch H, Kossen R, Suhr M, Freckmann L, Weber L, Henke C, Lehmann C, et al. Management of Metadata Types in Basic Cardiological Research. Stud Health Technol Inform. 2021:283:59-68.