Automated screening of research studies for systematic reviews using study characteristics

Guy Tsafnat, Paul Glasziou, George Karystianis, Enrico Coiera

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Background: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%. Methods: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. Results: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%. Conclusions: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

LanguageEnglish
Article number64
Pages1-9
Number of pages9
JournalSystematic Reviews
Volume7
Issue number1
DOIs
Publication statusPublished - 25 Apr 2018

Fingerprint

Research
Automation
Observational Studies
Population
Supervised Machine Learning

Bibliographical note

Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

  • Automation of systematic reviews
  • Evidence screening
  • Study characterisation
  • Study selection

Cite this

@article{f684082414704d94937b278f0069b56e,
title = "Automated screening of research studies for systematic reviews using study characteristics",
abstract = "Background: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50{\%}. Methods: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. Results: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7{\%} of retrieved studies with a recall of 98{\%}. Conclusions: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.",
keywords = "Automation of systematic reviews, Evidence screening, Study characterisation, Study selection",
author = "Guy Tsafnat and Paul Glasziou and George Karystianis and Enrico Coiera",
note = "Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",
year = "2018",
month = "4",
day = "25",
doi = "10.1186/s13643-018-0724-7",
language = "English",
volume = "7",
pages = "1--9",
journal = "Systematic Reviews",
issn = "2046-4053",
publisher = "Springer, Springer Nature",
number = "1",

}

Automated screening of research studies for systematic reviews using study characteristics. / Tsafnat, Guy; Glasziou, Paul; Karystianis, George; Coiera, Enrico.

In: Systematic Reviews, Vol. 7, No. 1, 64, 25.04.2018, p. 1-9.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Automated screening of research studies for systematic reviews using study characteristics

AU - Tsafnat, Guy

AU - Glasziou, Paul

AU - Karystianis, George

AU - Coiera, Enrico

N1 - Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2018/4/25

Y1 - 2018/4/25

N2 - Background: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%. Methods: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. Results: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%. Conclusions: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

AB - Background: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%. Methods: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. Results: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%. Conclusions: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

KW - Automation of systematic reviews

KW - Evidence screening

KW - Study characterisation

KW - Study selection

UR - http://www.scopus.com/inward/record.url?scp=85058913446&partnerID=8YFLogxK

U2 - 10.1186/s13643-018-0724-7

DO - 10.1186/s13643-018-0724-7

M3 - Article

VL - 7

SP - 1

EP - 9

JO - Systematic Reviews

T2 - Systematic Reviews

JF - Systematic Reviews

SN - 2046-4053

IS - 1

M1 - 64

ER -