An empirically defined decision tree to predict systematic reviews at risk of change in conclusion

Research output: Contribution to journalMeeting abstractResearch

Abstract

Background: Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice. Objectives: To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews. Methods: We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall. Results: We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months). Conclusions: An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.
LanguageEnglish
Pages122-123
Number of pages2
JournalCochrane Database of Systematic Reviews
Volume9
Issue numberSupplement 1
Publication statusPublished - Sep 2018
Event25th Cochrane Colloquium - Edinburgh, United Kingdom
Duration: 16 Sep 201818 Sep 2018

Fingerprint

Decision Trees
Research Personnel
Safety

Cite this

@article{4f8b9a68519c4aa08aa390854c3a7b06,
title = "An empirically defined decision tree to predict systematic reviews at risk of change in conclusion",
abstract = "Background: Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice. Objectives: To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews. Methods: We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall. Results: We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40{\%} and a recall of 70{\%}. After adding completeness to the decision tree, this increased to an average precision of 60{\%} and a recall of 90{\%}. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5{\%}), the original review had fewer trials (< 23) and more time had elapsed (> 53 months). Conclusions: An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.",
author = "R. Bashir and D. Surian and Dunn, {A. G.}",
year = "2018",
month = "9",
language = "English",
volume = "9",
pages = "122--123",
journal = "The Cochrane database of systematic reviews",
issn = "1469-493X",
publisher = "John Wiley & Sons",
number = "Supplement 1",

}

An empirically defined decision tree to predict systematic reviews at risk of change in conclusion. / Bashir, R.; Surian, D.; Dunn, A. G.

In: Cochrane Database of Systematic Reviews, Vol. 9, No. Supplement 1, 09.2018, p. 122-123.

Research output: Contribution to journalMeeting abstractResearch

TY - JOUR

T1 - An empirically defined decision tree to predict systematic reviews at risk of change in conclusion

AU - Bashir, R.

AU - Surian, D.

AU - Dunn, A. G.

PY - 2018/9

Y1 - 2018/9

N2 - Background: Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice. Objectives: To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews. Methods: We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall. Results: We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months). Conclusions: An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.

AB - Background: Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice. Objectives: To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews. Methods: We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall. Results: We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months). Conclusions: An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.

UR - https://doi.org/10.1002/14651858.CD201801

M3 - Meeting abstract

VL - 9

SP - 122

EP - 123

JO - The Cochrane database of systematic reviews

T2 - The Cochrane database of systematic reviews

JF - The Cochrane database of systematic reviews

SN - 1469-493X

IS - Supplement 1

ER -