Stopping criteria for prioritised screening
The stopping criteria offers a reliable method to decide when to stop screening during machine learning-prioritised screening for a systematic review, or for document review in ediscovery.
Such a method is vital in order to use machine learning safely (by estimating the risk of missing documents). Our method describes statistical confidence levels that any given level of recall has not been achieved. As such, it offers a robust data-driven method to decide when to stop, and a clear and transparent way of communicating the risks of stopping at any given point. Read vignette(“stopping-criteria”) for more details on ho this works in the package, or check out the original paper for a full description of the criteria, including theory and evaluations.
References & Source code
Buscar, short for Biased Urn based Stopping Criteria for technology Assisted Review, is an R and Python package implementing the stopping criteria described in
Callaghan, Max, and Finn Müller-Hansen. 2020. “Statistical Stopping Criteria for Automated Screening in Systematic Reviews.” Systematic Reviews. https://doi.org/10.21203/rs.2.18218/v2.