gms | German Medical Science

Introduction: In statistical consulting, incorporating advanced technologies like Large Language Models (LLMs), such as ChatGPT, offers promising enhancements to traditional methodologies. Integrating LLMs into daily work environments presents challenges across many areas. Dell’Acqua et al. [1] investigated how access to ChatGPT 4 impacts productivity and quality among consultants within a global management consulting company. Furthermore, ChatGPT's data analysis mode enables direct statistical analysis within the model itself, which introduces a wide range of new application areas. Irvine et al. [2] investigated opportunities and limitations of this plugin for hydrological analysis. However, both the use of LLMs as “consultant”, as well as “data analyst” face several risks [3], [4]. The aim of this study is to investigate the potential applications of LLMs within an academic statistical consulting context at the University Medical Center in Freiburg, particularly focusing on research-related tasks and data analysis. By assessing the utility, efficiency, and user satisfaction regarding the use of LLMs, this project aims to pave the way for more informed, efficient, and effective statistical consulting practices.

Methods: This study employs a mixed-method approach consisting of four parts, utilizing both qualitative and quantitative methods. While the fourth part represents a more general exploration related to the use of LLMs, this contribution focuses on the three first study parts addressing the statistical consulting setting:

Study Part I: a combination of semi-structured interviews and a longitudinal prospective standardized online questionnaire to gather insights from consultants on the use of LLMs.

Study Part II: evaluation of a specifically developed training module for consultants through a standardized online questionnaire, assessing the module’s effectiveness and areas for improvement.

Study Part III: collection of feedback from advisees via a standardized questionnaire, focusing on their experiences and satisfaction with LLM-enhanced consulting sessions.

Results: We will discuss the specific design and considerations behind the study in detail and share insights from our experiences encountered during the execution. As the study will be ongoing by the time of the conference, we will present preliminary results. These will include reactions and feedback from the training modules, as along with descriptions of the collected data from study parts I and II. Preliminary results on consultant and advisee satisfaction, perceived utility, and potential operational challenges with LLMs will be reported, aiming to provide insights into the acceptance and practical implementation of LLMs in our consulting practices.

Conclusion: The integration of LLMs into an academic statistical consulting service at the University Medical Center in Freiburg holds considerable promise for transforming traditional practices. This ongoing study aims to illustrate the roles that LLMs can play in improving the efficiency and effectiveness of statistical services. Initial findings from the EXPOLS study are expected to guide the development of targeted training programs, optimize LLM functionalities within statistical consulting, and ultimately foster a more efficient and collaborative research environment. Continuous evaluation will ensure that these technological advancements meet the evolving needs of academic research communities.

The authors declare that they have no competing interests.

The authors declare that a positive ethics committee vote has been obtained.