Stopping criteria for prioritised screening
The stopping criteria offers a reliable method to decide when to stop screening during machine learning-prioritised screening for a systematic review, or for document review in ediscovery.
Such a method is vital in order to use machine learning safely (by estimating the risk of missing documents). Our method describes statistical confidence levels that any given level of recall has not been achieved. As such, it offers a robust data-driven method to decide when to stop, and a clear and transparent way of communicating the risks of stopping at any given point. Read vignette(“stopping-criteria”) for more details on ho this works in the package, or check out the original paper for a full description of the criteria, including theory and evaluations.
References & Source code
Buscar, short for Biased Urn based Stopping Criteria for technology Assisted Review, is an R and Python package implementing the stopping criteria described in
Callaghan, Max, and Finn Müller-Hansen. 2020. “Statistical Stopping Criteria for Automated Screening in Systematic Reviews.” Systematic Reviews. https://doi.org/10.21203/rs.2.18218/v2.
- BuscarR R package
- BuscarPy Python package
Max Callaghan
Researcher
Finn Müller-Hansen
Researcher
Finn Müller-Hansen works in the group “Applied Sustainability Science” (APSIS) on applications of machine learning and bibliometric analysis on the scientific literature and public as well as political discourse about energy transitions. He is interested in methods from complex systems theory to investigate the interplay of social, economic, and ecological factors in transformations towards a sustainable economy.