Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?

Ying Wang, Enrico Coiera, Farah Magrabi*

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Objective: The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity.

Materials and methods: Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type = 2860, n_severity = 1160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent =6000/5950).

Results: SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics/BOW]: benchmark = 82.6%/69.4%; original = 77.9%/68.8%; independent = 78.0%/67.4%) and extreme-risk events (F-score [semantics/BOW]: benchmark = 87.3%/87.3%; original = 25.5%/19.8%; independent = 49.6%/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics/BOW]: benchmark = 48.9%/39.9%; original = 57.9%/44.4%; independent = 59.5%/34.9%).

Discussion: BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance.

Conclusions: UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.
Original languageEnglish
Number of pages9
JournalJournal of the American Medical Informatics Association : JAMIA
DOIs
Publication statusE-pub ahead of print - 18 Jun 2020

Keywords

  • UMLS
  • semantics
  • patient safety
  • incident reporting
  • supervised machine learning
  • text classification
  • natural language processing

Fingerprint Dive into the research topics of 'Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?'. Together they form a unique fingerprint.

  • Cite this