gms | German Medical Science

Introduction: For medical research questions, machine learning (ML) has become a commonly used tool [1]. Due to various methodological pitfalls, an appropriate development and validation of ML models might be difficult for non-experts [2], [3]. Interactive guidance based on expert knowledge may assist practitioners with these tasks but is not readily available so far. We aim to fill this gap with our ML guidance toolkit mlguide, a novel expert system to provide methodological support in applied ML [4]. Our goal is to recommend suitable methods that are tailored to the users ML task based on the current (published) evidence. The development of the mlguide toolkit is part of the KI-FDZ project that aims at facilitating the usage of artificial intelligence at the Health Data Lab [5].

Methods: We conducted a literature review to derive a workflow suitable for supervised ML models. Second, an online survey was created to determine which problems are most common and important during ML model development. Furthermore, we developed a first software concept of our mlguide toolkit and its architecture.

Results: Our ML workflow consists of four parts: problem definition – data processing – model development – deployment. The mlguide toolkit will support the second and third part interactively by providing methodological recommendations based on the user-specified problem description.

The survey targeting ML practitioners, data scientists and students, among others, has been completed and is now accessible online (http://websites.fraunhofer.de/mlguide-user-survey/index.php/578345).

Based on a literature review about expert systems, we designed the architecture of mlguide toolkit with three components: The web application mlguide.app provides an interface for users to interact with the whole system. Its backend is connected to the R package mlguide which contains all development tools and the guidance engine for managing user requests for evidence-based recommendations. For this purpose, mlguide interacts with mlguide.core that contains and manages the expert knowledge.

Discussion: The current toolkit version addresses only supervised ML models and tabular data. Several extensions to other ML models or data types are thus possible. The results of our open survey will guide our future developments of the supported ML workflow and our web application. An early prototype of the mlguide toolkit is currently running and being tested internally.

Conclusion: We presented the current state of our mlguide toolkit that guides ML practitioners during the development of a ML model. It supports the user with the data preparation and model development process. In future work, the results of the first user survey will be used to improve our current ML workflow and our current mlguide toolkit implementation. One important next implementation step will be to optimize the process of capturing evidence in the system by ML experts and to explore how unstructured knowledge based on publications can be extracted in a suitable structured format. Furthermore, we will conduct user interviews with more specific questions to determine further potential improvements.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.