Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?

Ying Wang, Enrico Coiera, Farah Magrabi*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)
37 Downloads (Pure)


Objective: The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity.

Materials and methods: Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type = 2860, n_severity = 1160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type/severity: n_benchmark = 286/116, n_original = 444/4837, n_independent =6000/5950).

Results: SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics/BOW]: benchmark = 82.6%/69.4%; original = 77.9%/68.8%; independent = 78.0%/67.4%) and extreme-risk events (F-score [semantics/BOW]: benchmark = 87.3%/87.3%; original = 25.5%/19.8%; independent = 49.6%/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics/BOW]: benchmark = 48.9%/39.9%; original = 57.9%/44.4%; independent = 59.5%/34.9%).

Discussion: BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance.

Conclusions: UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.
Original languageEnglish
Pages (from-to)1502-1509
Number of pages8
JournalJournal of the American Medical Informatics Association : JAMIA
Issue number10
Early online date18 Jun 2020
Publication statusPublished - 1 Oct 2020

Bibliographical note

Copyright the Author(s) 2020. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.


  • UMLS
  • semantics
  • patient safety
  • incident reporting
  • supervised machine learning
  • text classification
  • natural language processing


Dive into the research topics of 'Can Unified Medical Language System-based semantic representation improve automated identification of patient safety incident reports by type and severity?'. Together they form a unique fingerprint.

Cite this