Evaluating the influence of data collector training for predictive risk of death models

an observational study

Arvind Rajamani*, Stephen Huang, Ashwin Subramaniam, Michele Thomson, Jinghang Luo, Andrew Simpson, Anthony McLean, Anders Aneman, Thodur Vinodh Madapusi, Ramanathan Lakshmanan, Gordon Flynn, Latesh Poojara, Jonathan Gatward, Raju Pusapati, Adam Howard, Debbie Odlum

*Corresponding author for this work

Research output: Contribution to journalArticle


Background: Severity-of-illness scoring systems are widely used for quality assurance and research. Although validated by trained data collectors, there is little data on the accuracy of real-world data collection practices. Objective: To evaluate the influence of formal data collection training on the accuracy of scoring system data in intensive care units (ICUs). Study design and methods: Quality assurance audit conducted using survey methodology principles. Between June and December 2018, an electronic document with details of three fictitious ICU patients was emailed to staff from 19 Australian ICUs who voluntarily submitted data on a web-based data entry form. Their entries were used to generate severity-of-illness scores and risks of death (RoDs) for four scoring systems. The primary outcome was the variation of severity-of-illness scores and RoDs from a reference standard. Results: 50/83 staff (60.3%) submitted data. Using Bayesian multilevel analysis, severity-of-illness scores and RoDs were found to be significantly higher for untrained staff. The mean (95% high-density interval) overestimation in RoD due to training effect for patients 1, 2 and 3, respectively, were 0.24 (0.16, 0.31), 0.19 (0.09, 0.29) and 0.24 (0.1, 0.38) respectively (Bayesian factor >300, decisive evidence). Both groups (trained and untrained) had wide coefficients of variation up to 38.1%, indicating wide variability. Untrained staff made more errors in interpreting scoring system definitions. Interpretation: In a fictitious patient dataset, data collection staff without formal training significantly overestimated the severity-of-illness scores and RoDs compared with trained staff. Both groups exhibited wide variability. Strategies to improve practice may include providing adequate training for all data collection staff, refresher training for previously trained staff and auditing the raw data submitted by individual ICUs. The results of this simulated study need revalidation on real patients.

Original languageEnglish
Article number2020-010965
JournalBMJ Quality and Safety
Early online date30 Mar 2020
Publication statusE-pub ahead of print - 30 Mar 2020
Externally publishedYes


  • critical care
  • healthcare quality improvement
  • quality measurement

Fingerprint Dive into the research topics of 'Evaluating the influence of data collector training for predictive risk of death models: an observational study'. Together they form a unique fingerprint.

  • Cite this

    Rajamani, A., Huang, S., Subramaniam, A., Thomson, M., Luo, J., Simpson, A., ... Odlum, D. (2020). Evaluating the influence of data collector training for predictive risk of death models: an observational study. BMJ Quality and Safety, [2020-010965]. https://doi.org/10.1136/bmjqs-2020-010965