A study on explainable AI in healthcare: a brief report

Despite its exponential growth, artificial intelligence (AI) in healthcare faces various challenges. One of the problems is a lack of transparency and explainability around healthcare AI. This arguably leads to insufficient trust in AI technologies, quality, and accountability and liability issues. In our pilot study we examined whether, why, and to what extent AI explainability is needed with relation to AI-enabled medical devices and their outputs. Relying on a critical analysis of interdisciplinary literature on this topic and a pilot empirical study, we conclude that the role of technical explainability in the medical AI context is a limited one. Technical explainability is capable to addresses only a limited range of challenges associated with AI and is likely to reach fewer goals than sometimes expected. The study shows that, instead of technical explainability of medical AI devices, most stakeholders need more transparency around its development and quality assurance process.