Applying LLMs for Analysis of AI/ML Medical Device Approvals and Safety Events

Medical devices are increasingly integrating AI including machine learning (ML) algorithms, yet their clinical roles and associated safety risks remain poorly understood. Up to 16% of AI safety events reported to the US Food and Drug Administration (FDA) involved patient harm or death. Manual analysis of FDA approval documents to extract crucial information (e.g., ML functions, clinical applications) is time-consuming and prone to human error. Moreover, the black-box nature of many AI systems hinders the identification of unique safety risks.

This project investigates the feasibility of leveraging large language models (LLMs) to automate the analysis of two FDA data sources: 1/ ML medical device approvals; and 2/ reports about AI safety events, enabling more efficient and comprehensive safety assessment.