Skip to main content

Table 8 Characteristics of stage II evaluation of methods studies

From: Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction

First Author Year
Title
Primary objective Existence of a protocol Study design Health field the filter designed for # of filters evaluated (# filters developed by the author) Database (interfaces) Technique to identify and/or create a gold standard Sample size of the gold standard set or validation set (n) Validation on a data set distinct from the derivation data Performance measures used Search dates for the gold standard or validation set Name of filters evaluated (number of filters)
Boluyt 2008 [13]
Usefulness of systematic review search strategies in finding child health systematic reviews in MEDLINE.
Assess search filters for child health SRs in PubMed NR Comparative search filter evaluation Child health 9 PubMed Handsearching, Developed based on database searches 387 Yes Sensitivity/recall, precision Handsearch 1994, 1997, 2000, 2002, and 2004; DARE up to 2004, and year 2006 PubMed filter 2006
Shojania 2001
Boynton 1998
White 2001 (two)
Montori 2005 (four)
Boynton 1998 [15]
Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design.
Evaluate propose a range of search strategies to identify SRs in MEDLINE NR Search filter evaluation, Comparative search filter evaluation Medicine (general and internal) 15 (11) MEDLINE (Ovid) Handsearching 288 No Sensitivity/recall, precision 1992 and 1995 Boynton 1998 (eleven)
Hunt 1997 (two)
CRD - Oxman 1994 (two)
Eady 2008 [19]
PsycINFO search strategies identified methodologically sound therapy studies and review articles for use by clinicians and researchers
Evaluate search strategies for finding SRs in PsycINFO NR Search filter evaluation Psych. N/A PsycINFO Handsearching 58 No Sensitivity/recall, precision, specificity, accuracy 2000 Eady 2008
Golder 2006 [20]
Identifying systematic reviews of the adverse effects of health care interventions.
Identify SRs of adverse effects in two major databases NR Search filter evaluation Adverse effects N/A DARE (CDSR and CRD) Developed based on database searches 270 No Sensitivity/recall, precision 1994 to 2005 Golder 2006
Lee 2012 [16]
An optimal search filter for retrieving systematic reviews and meta-analyses.
Develop and validate the health-evidence.ca SR filter and compare its performance to other filters NR Search filter evaluation, Comparative search filter evaluation Public health 31 (3) MEDLINE, EMBASE, and CINAHL Handsearching, Developed based on database searches 219 Yes Sensitivity/recall, precision, specificity NNR 2004/2005 health-evidence.ca SR filter - Lee 2012 (three)
Montori [2005- four]
Hunt 1997 (two)
Shojania 2001
Boynton 1998 (two)
BMJ Clin Evidence n.d.
CRD Ciliska 2007 (four)
SIGN n.d.
Wilczynski 2007 (four)
McKibbon 1998
Montori 2005 [17]
Optimal search strategies for retrieving systematic reviews from Medline: analytical survey.
Develop optimal search strategies in Medline for retrieving SRs NR Search filter evaluation, Comparative search filter evaluation Medicinefamily practice, nursing,
mental health
10 (4) MEDLINE Handsearching 753 Yes Sensitivity/recall, specificity, precision 2000 Montori 2005 (four)
White 2001 (three)
Hunt 1997 (two)
Shojania 2001
Rathbone 2016 [12]
A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension.
Evaluate seven databases to determine their coverage of SRs of hypertension NR Comparative database evaluation Hypertension N/A Cochrane, DARE, EMBASE, Epistemonikos, MEDLINE, PubMed, and TRIP Developed based on database searches 440 N/A Sensitivity/recall, precision 2003–2015 SR filters incorporated into the databases; MEDLINE used Montori 2005
Shojania 2001 [21]
Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy.
Evaluate a search strategy for identifying SRs NR Search filter evaluation Treatment diagnosis, prognosis, causation, quality improvement, or economics N/A MEDLINE (PubMed) Handsearching, Developed based on database searches 104 No Sensitivity/recall, precision 1999–2000 PubMed n.d.
White 2001 [18]
A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy.
Improve methods to derive a more objective search strategy to identify SRs in MEDLINE NR Search filter evaluation, Comparative search filter evaluation Treatment diagnosis, prognosis, causation 7 (5) MEDLINE (Ovid) Handsearching journals 110 No Sensitivity/recall, precision 1995 and 1997 White 2001 (five]
Boynton 1998
CRD - Wolf 1996
Wilczynski 2007 [22]
EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews.
Develop search strategies that optimize the retrieval of SRs from EMBASE. NR Search filter evaluation Internal medicinegeneral practice, mental health, nursing practice N/A MEDLINE Handsearching journals 220 No Sensitivity/recall, specificity, precision, accuracy 2000 Wilczynski 2007
Wilczynski 2009 [23]
Consistency and accuracy of indexing systematic review articles and meta-analyses in medline.
Determine the consistency and accuracy of indexing SRs and meta-analyses in MEDLINE NR Search filter evaluation Medicine N/A MEDLINE Developed based on database searches NA No Sensitivity/recall, specificity, precision, accuracy 2000 Wilczynski 2009
Wilczynski 2011 [24]
Sensitive Clinical Queries retrieved relevant systematic reviews as well as primary studies: an analytic survey.
Determine how well the previously validated broad and narrow Clinical Queries retrieve SRs NR Search filter evaluation Therapy, diagnosis prognosis, etiology N/A MEDLINE, EMBASE, CINAHL, and PsycINFO Developed based on database searches NA No Sensitivity/recall, specificity precision 2000 Wilczynski 2011
Wong 2006 [14]
Comparison of top-performing search strategies for detecting clinically sound treatment studies and systematic reviews in MEDLINE and EMBASE.
Compare sensitivity and specificity of search strategies for detecting reviews in MEDLINE and EMBASE NR Comparative search filter evaluation Medicine 7 MEDLINE, EMBASE Handsearching journals 753 in MEDLINE, 220 in EMBASE N/A Sensitivity/recall, specificity precision 2000 Montori 2005 (three)
Wilczynski 2007 (four]
Wong 2006 [26]
Optimal CINAHL search strategies for identifying therapy studies and review articles.
Design optimal search strategies for locating review articles in CINAHL NR Search filter evaluation Nursing and allied health N/A CINAHL Handsearching journals 127 No Sensitivity/recall, specificity precision, accuracy 2000 Wong 2006
Zacks 1998 [25]
Developing search strategies for detecting high quality reviews in a hypertext test collection.
Determine whether sensitive and specific search strategies exist to select SRs NR Search filter evaluation Etiology, prognosis, therapy diagnosis N/A SWISH v.1.1.1 Developed based on database searches 209 No Sensitivity/recall, specificity Not reported Zacks 1998
  1. Sensitivity/recall is defined as the proportion of relevant reports correctly retrieved by the filter; Precision is the number of relevant reports retrieved divided by the total number of records retrieved by the filter; NNR is the inverse of the precision; Specificity is the proportion of irrelevant reports correctly not retrieved by the filter, Accuracy is the proportion of all reports that are correctly classified
  2. CDSR Cochrane Database of Systematic Reviews, CRD Centre for Review and Dissemination, n.d. no date, NNR number needed to read, N/A not applicable, NR not reported, PH public health, SRs systematic reviews, SWISH Simple Web Indexing System for Humans