<div class="csl-bib-body">
<div class="csl-entry">Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023). An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews. <i>Intelligent Systems with Applications</i>, <i>18</i>, Article 200193. https://doi.org/10.1016/j.iswa.2023.200193</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/188936
-
dc.description.abstract
Citation screening is an essential and time-consuming step of the systematic literature review process in medicine. Multiple previous studies have proposed various automation techniques to assist manual annotators in this tedious task. The most widely used measure for the evaluation of automated citation screening techniques is Work Saved over Sampling (WSS). In this work, we analyse this measure and examine its drawbacks. We subsequently propose to normalise WSS which enables citation screening performance comparisons across different systematic reviews. We analytically show that normalised WSS is equivalent to the True Negative Rate (TNR). Finally, we provide benchmark scores for fifteen systematic review datasets with TNR@95% recall measure and compare the measure with Precision and AUC.
en
dc.description.sponsorship
European Commission
-
dc.language.iso
en
-
dc.publisher
Elsevier
-
dc.relation.ispartof
Intelligent Systems with Applications
-
dc.subject
Citation screening
en
dc.subject
Evaluation
en
dc.subject
Study selection
en
dc.subject
systematic reviews
en
dc.subject
Work Saved over Sampling (WSS)
en
dc.subject
WSS
en
dc.title
An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews