Mapping the flow of knowledge as guidance for ethics implementation in medical AI: a qualitative study

In response to the COVID-19 crisis, Artificial Intelligence (AI) has been applied to a range of applications in healthcare and public health such as case identification or monitoring of the population. The urgency of the situation should not be to the detriment of considering the ethical implications of such apps. Implementing ethics in medical AI is a complex issue calling for a systems thinking approach engaging diverse representatives of the stakeholders in a consultative process. The participatory engagement aims to gather the different perspectives of the stakeholders about the app in a transparent and inclusive way. In this study, we engaged a group of clinicians, patients, and AI developers in conversations about a fictitious app which was an aggregate of actual COVID-19 apps. The app featured a COVID-19 symptoms monitoring function for both the patient and the clinician, as well as infection clusters tracking for health agencies. Anchored in Soft Systems Methodology and Critical Systems Thinking, participants were asked to map the flow of knowledge between the clinician, the patient, and the AI app system and answer questions about the ethical boundaries of the system. Because data and information are the resource and the product of the AI app, understanding the nature of the information and knowledge exchanged between the different agents of the system can reveal ethical issues. In this study, not only the output of the participatory process was analysed, but the process of the stakeholders' engagement itself was studied as well. To establish a strong foundation for the implementation of ethics in the AI app, the conversations among stakeholders need to be inclusive, respectful and allow for free and candid dialogues ensuring that the process is transparent for which a systemic intervention is well suited.