The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Objectives: To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates. Study Design and Setting: We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC). Results: We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39). Conclusion: When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

LanguageEnglish
Pages42-49
Number of pages8
JournalJournal of Clinical Epidemiology
Volume110
DOIs
Publication statusPublished - 1 Jun 2019

Fingerprint

Area Under Curve
Learning
Databases
ROC Curve
Machine Learning

Keywords

  • Automation of systematic reviews
  • Classification trees
  • Clinical trial registries
  • Machine learning
  • Systematic reviews as topic
  • Updating systematic reviews

Cite this

@article{b9a85b6a333346f584450320e304bf1d,
title = "The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples",
abstract = "Objectives: To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates. Study Design and Setting: We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC). Results: We identified 63 pairs of systematic reviews and updates, of which 20 (32{\%}) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39). Conclusion: When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.",
keywords = "Automation of systematic reviews, Classification trees, Clinical trial registries, Machine learning, Systematic reviews as topic, Updating systematic reviews",
author = "Rabia Bashir and Didi Surian and Dunn, {Adam G.}",
year = "2019",
month = "6",
day = "1",
doi = "10.1016/j.jclinepi.2019.02.015",
language = "English",
volume = "110",
pages = "42--49",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier",

}

TY - JOUR

T1 - The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples

AU - Bashir, Rabia

AU - Surian, Didi

AU - Dunn, Adam G.

PY - 2019/6/1

Y1 - 2019/6/1

N2 - Objectives: To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates. Study Design and Setting: We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC). Results: We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39). Conclusion: When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

AB - Objectives: To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates. Study Design and Setting: We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC). Results: We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39). Conclusion: When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

KW - Automation of systematic reviews

KW - Classification trees

KW - Clinical trial registries

KW - Machine learning

KW - Systematic reviews as topic

KW - Updating systematic reviews

UR - http://www.scopus.com/inward/record.url?scp=85063330095&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2019.02.015

DO - 10.1016/j.jclinepi.2019.02.015

M3 - Article

VL - 110

SP - 42

EP - 49

JO - Journal of Clinical Epidemiology

T2 - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -