gms | German Medical Science

Introduction: Despite the continuous improvements in performance, especially in clinical contexts, a major pitfall of AI-DSS remains their epistemic opacity [1], [2], [3]. Thus, the handling of supposedly unexplainable technology in healthcare is an active field of research for both technical and ethical scholars. This work presents a mapping review of the literature on the normativity of explainability of AI in healthcare, with the goal to provide an overview of the current debate for ethicists, developers, and practitioners of AI-DSS [4], [5]. To do so, we provide a case study to incorporate the findings into the AI-DSS development process.

Methods: We conducted a literature search on PubMed, BASE, and Scopus for English documents between 2016 and 2023. The inclusion criterion was to discuss the normativity of at least one notion of explainability as “explainability”, “explicability”, “interpretability”, “contestability”, “transparency”, “black box” or “understandability”. Surveys, systematic reviews, and technical or empirical investigations in explainability were excluded from the review. The epistemic assumptions, normative motivation, and normative and technical requirements outlined in the documents were qualitatively categorized and demonstrated in a case study.

Results The literature search resulted in 1320 documents. After merging duplicates, we obtained 882 documents, of which 80 documents were found relevant in an abstract screening.

First, we find that the normative requirements on explainability of AI in healthcare can be categorized according to (a) their underlying epistemic assumptions on the decision-making process of AI-DSS and clinical experts, and (b) the normative stance and thus, perspective, taken by the respective scholars.

Second, we identified three major positions on the normativity of explainability in healthcare.

1.

A lack of explainability is not morally justifiable, AI-DSS must be explainable.

2.

A lack of explainability is morally justifiable, AI effectivity must be sufficiently validated.

3.

AI-DSS must be at least as explainable as the human counterpart in decision-making, its effectivity needs to be sufficiently validated.

Discussion As an important source of disagreement on the normative judgments on explainability of AI in healthcare, epistemological assumptions on AI and human explanations were identified. Given that, empirical research on the matter may inform the debate better and be able to resolve the foundational disagreements. However, normative stances will remain a source of disagreement and must be considered when deploying AI-DSS in healthcare.

Moreover, a mapping of existing solutions to the identified normative requirements of explainability would be desirable and might identify specific research gaps for AI-DSS developers in healthcare.

Conclusion The presented mapping review informs developers, practitioners, and scholars of ethics alike. The identified epistemic assumptions may form a research agenda to inform the debate with well-founded empirical research. To ethicists, this review may provide an overview of the current state of the debate. Developers and practitioners in the field may use this review to get an overview of recommended actions given different normative stances and applications. Finally, this review may help translate normative requirements into policies and technological requirements.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.