Applying LLMs for analysis of AI/ML medical device approvals

Machine learning (ML) is increasingly being integrated into medical devices. However, the functionality and clinical use of ML within these devices remains unclear, raising safety concerns. Manual analysis of FDA approval documents provides insights but is inefficient. This study explores the feasibility of using large language models (LLMs) to automate such analyses. We evaluate LLMs based on architecture, training strategies, parameter sizes, computational demands, and output quality to extract device characteristics, ML functions, and clinical applications. Analyzing 108 approvals, we found that decoder LLMs effectively extracted explicit details but are computationally intensive, whereas encoder models infer clinical context more efficiently. All models require domain-specific optimization for accurate ML-related extraction.