The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews
© Bramer et al.; licensee BioMed Central Ltd. 2013
Received: 12 November 2013
Accepted: 13 December 2013
Published: 23 December 2013
The usefulness of Google Scholar (GS) as a bibliographic database for biomedical systematic review (SR) searching is a subject of current interest and debate in research circles. Recent research has suggested GS might even be used alone in SR searching. This assertion is challenged here by testing whether GS can locate all studies included in 21 previously published SRs. Second, it examines the recall of GS, taking into account the maximum number of items that can be viewed, and tests whether more complete searches created by an information specialist will improve recall compared to the searches used in the 21 published SRs.
The authors identified 21 biomedical SRs that had used GS and PubMed as information sources and reported their use of identical, reproducible search strategies in both databases. These search strategies were rerun in GS and PubMed, and analyzed as to their coverage and recall. Efforts were made to improve searches that underperformed in each database.
GS’ overall coverage was higher than PubMed (98% versus 91%) and overall recall is higher in GS: 80% of the references included in the 21 SRs were returned by the original searches in GS versus 68% in PubMed. Only 72% of the included references could be used as they were listed among the first 1,000 hits (the maximum number shown). Practical precision (the number of included references retrieved in the first 1,000, divided by 1,000) was on average 1.9%, which is only slightly lower than in other published SRs. Improving searches with the lowest recall resulted in an increase in recall from 48% to 66% in GS and, in PubMed, from 60% to 85%.
Although its coverage and precision are acceptable, GS, because of its incomplete recall, should not be used as a single source in SR searching. A specialized, curated medical database such as PubMed provides experienced searchers with tools and functionality that help improve recall, and numerous options in order to optimize precision. Searches for SRs should be performed by experienced searchers creating searches that maximize recall for as many databases as deemed necessary by the search expert.
KeywordsBibliographic databases Information retrieval Systematic reviews Methodology Literature searching Reproducibility
For several years, information specialists have discussed which databases (and how many) should be used to perform exhaustive searches of the literature. Prior to 2004, the year of Google Scholar’s release, these discussions focused primarily on traditional databases such as Embase and MEDLINE [1, 2]. Further, the general consensus had developed that searching a limited number of databases was insufficient where completeness was the goal [3–6].
In addition, the type of searching that is required to support systematic reviews (SRs) is more complex and time-consuming than searching for simple clinical queries. The demands placed on a searcher for the SR are much higher than for other searches because of the specific requirements of the SR  which is integral to science and must therefore be performed systematically, and made repeatable, verifiable and accountable [8, 9].
Since 2004, Google Scholar (GS) has been widely-used to locate specific items and aid in cumulating the scholarly literature. In 2005, Giustini  stated that GS produced acceptable results for browsing routines but results of low precision meant that its use for other searching was problematic. Since then, GS has improved its scope; from 2005 to 2012, its coverage of the literature rose from 30 to 88% to 98 to 100% [11, 12]. An important unanswered question about GS remains: 'Is GS advanced enough in its development to replace more sophisticated tools such as PubMed or Embase?’
In 2007, Shultz  provided an overview of criticism of GS that had been generated since its debut in 2004. Unfortunately, many of the original shortcomings identified between GS and traditional bibliographic databases such as MEDLINE and Embase are still in evidence: GS lacks a controlled vocabulary, search histories and sets cannot be built and manipulated and wildcards and limits (for instance study types) cannot be used precisely. Only the first 1,000 citations of any search in GS are viewable and search strings must be kept under 256 characters.
Since 2004, a number of studies have examined the value of GS in biomedical searching. Falagas, Pitsouni, et al. compared four databases including GS and PubMed, and concluded that GS retrieved more obscure items than other search tools. Anders and Evans  focused on using advanced searching in GS and PubMed but, given major differences in the databases, their study made true comparisons difficult. In Nourbakhsh, Nugent, et al., researchers found that the first 20 results in GS often produced more relevant hits than similar searches in PubMed. But since PubMed, until very recently, lists citations in chronological order (not by algorithms, as in GS) the authors’ conclusions are counter-intuitive. The frequently cited study by Walters  covered one topic (on older person migration), which is out-of-scope in a medical database. A recent study by Shariff, Bejaimal, et al. compared search strategies designed by end users and compared the first 40 hits in GS and PubMed.
In 2013, Gehanno, Rollin, et al. published a paper that generated important critical discussion of the value of GS. In their paper, Gehanno et al. used GS to locate all studies originally cited in a published SR. By finding all known items, the authors argued that GS, after some improvements to increase its search precision, could be used alone in searching for SRs.
The article by Gehanno et al. drew much attention to GS and resulted in some follow-up articles. Giustini and Kamel Boulos  argued that a 'known-item’ searching is a very different activity than locating articles by subject, as is attaining 100% recall of the (mostly unknown) relevant literature. Boeker, Vach, et al. reinvestigated the results of Gehanno et al. using search strategies designed to match Medline strategies used for Cochrane systematic reviews. However, the authors designed the searches themselves and failed to account for the maximum number of results that can be retrieved in GS (1,000), although they mentioned this limitation in their manuscript. The low precision in GS as reported by Gehanno et al. and by Boeker et al., is mainly an artifact due to the large number of hits that are reported by GS. Since search results cannot be viewed beyond the first 1,000 references, actual precision in GS should be calculated as the number of relevant references found in those first 1,000 references, divided by the number of hits that can actually be viewed, which is 1,000 at most.
Limitations of current published research on the usability of Google Scholar for medical purposes
Limitations of research
Not testing for systematic reviews
Limited number of searches
Relevancy of results only determined by the authors
Not reviewing the first 1,000 hits in Google Scholar
Only using searches designed by the authors
Searches not comparable between the databases
Published more than five years ago
Only searching for known items
Only looking at coverage, not retrieval
Though it seems unlikely that an experienced information specialist would use GS as the sole database in a SR, a less experienced researcher, faced with the enormous task of performing a review without expert help, might be tempted to do so (based on the aforementioned research). At least one review is known that, after doing preliminary searches in a wide range of databases decided ultimately to use only PubMed and GS, but failed to notice their search strategy was not executable in GS, since it was over 500 characters long .
In this paper, the usability of GS in searching for SRs is considered, where relevancy is pre-determined by inclusion in papers that have been previously published. Both the original (identical) topical searches reported by those papers and searches improved by an information specialist are used in order to compare the recall within GS and PubMed. The aim of this paper is to discover whether the original authors would have found all included references by using GS only. When studies from the original SRs were not found, it is assessed whether a more exhaustive search strategy created by an information specialist would improve recall. Given its potential for one-stop searching, we assess whether GS can indeed replace the multiple databases required for the SR and locate all studies needed to conduct a SR.
In May 2013, PubMed and Embase were searched using the exact phrases 'systematic review’ and 'google scholar’ in title and/or abstract fields. Of the records identified, the full-text of relevant papers was retrieved on the open web or via subscriptions at the first author’s institution. The full-text and appendices of available articles were scanned for descriptions of the strategies used to search PubMed and GS.
Inclusion and exclusion criteria
Systematic review in a medical topic
Length of search strategy greater than 256 characters
Reporting the use of both Google Scholar and PubMed
Number of hits retrieved in PubMed exceeds the reported total number of hits reviewed
Reporting in reproducible detail an identical single phrase search for these databases
Searches that were reproducible were executed in both GS and PubMed, and the number of hits was documented accordingly. In PubMed, results were limited to before the MeSH date (field: [mhda]) of the original search date as stated in the article. The number of hits in PubMed was compared with the number originally reported (either for PubMed, or the total for all databases). If it did not exceed that number, the SR was included. In GS, search results were limited to the publication start year used by the original authors and the end publication year of the search date. Because publication dates can differ from search dates (because publication dates are generally added to the print version, while the electronic version might be available longer) we checked whether the list of includes contained articles with newer publication dates, and if so, changed the publication limits accordingly.
For each replicated search, the first 1,000 results of GS were saved in a Word document. Using the 'find’ function in Word, occurrences of each included reference from the original SR were identified. Distinctive fragments of the title were searched but where no match was located, author names were searched. If a citation was not found among the first 1,000 results, GS coverage for that item was checked using author names and part of the title between double quotes. If the item was indeed present in GS, the reference was checked for retrieval in the search query (beyond the first 1,000 hits) by combining author names and distinctive title words with the full query (to check whether they had ranked low on Google’s PageRank algorithm).
We did not exclude hits that were citations only (by definition) but for those references, it was checked whether the citing articles, as linked in GS, were published before the citing systematic review. It could then be assumed that the citation was present in GS when the original authors performed their searches. If the citation was only present in articles with a more recent publication year, the result was confidently discarded.
All included studies were searched in PubMed by searching for the complete reference. If PubMed did not reveal a match, a second attempt was performed using a combination of first author [1au], page number [pg] and publication year [dp]. Included references were collected using the PubMed Clipboard. Once all references were retrieved, clipboard contents were checked against the results of the replicated search.
The intention of this project was not to judge the quality of the replicated searches. In a later stage, we improved some searches to investigate whether more citations could be found. An experienced information specialist (WB) created improved search strategies for GS based on the original authors’ description of their research question, without taking into account the included references from that SR. A second search strategy was designed based on the frequency of words in the titles of included references of these SRs. For the searches that had missed the most included references in PubMed, an information specialist created a more comprehensive search strategy using MeSH terms and free text, without using the included references to determine search words.
In 21 articles, the cited searches for both GS and PubMed were identical, well-documented, and the number of hits in PubMed did not exceed the number of hits first reported. Additional file 1 describes the original and replicated searches along with other parameters and the detailed results. In eleven cases, the searches used in this research were exactly the same as those described in the full text or appendices of the studies. In ten instances, some minor changes had to be made. In some cases, Boolean operators (AND and OR) were not stated and nesting was not clearly laid out; proper searches cannot be performed without operators but what was intended was clear. If major changes were required, the reviews had been excluded. For five references retrieved as citations, the citing articles all had a publication date later than the original search date, so these citations were ignored. A full list of references to all SRs included in this article can be found in Additional file 2.
The total number of studies included by the SRs was 541. In GS, ten studies were not present, thus the overall coverage of GS reached 98%. In PubMed, 48 references were not present, so PubMed had an overall coverage of 91%.
Of the total number of included studies that were reviewed (541), 389 (72%) were present in the first 1,000 hits of the original searches in GS. Forty-five articles had been retrieved by the search strategy in GS, but were not among the first 1,000 hits. If GS had allowed its users to review all search results, recall would thus have been 80%. The same searches retrieved 369 hits in PubMed (68%).
Improvement of search strategies in Google Scholar
Systematic reviews of which more than ten included references were not retrieved in Google Scholar; performance of improved searches
Number of included references
Number of includes retrieved by
Improved search #1
Improved search #2
Improvement of search strategies in PubMed
Systematic reviews that contributed most to the 'not retrieved articles’ in PubMed; performance of improved searches
Number of included references
Number of includes retrieved by
Literature searching in multiple databases can often be cumbersome and is always time-consuming if it is done well. GS offers an easy-to-use, familiar interface and relevance ranking, making simple searching for a few good articles much easier. However, the use of GS as a robust search tool is not without its challenges.
To focus on the differences between the databases, the research was restricted to SRs that used both GS and PubMed with identical search strategies. Although this research represents a small sample of all published SRs, and is not representative, we nonetheless believe our findings to be of indicative of a trend. Had SRs been selected that did not describe their GS search strategies, it would have been necessary to create them which was not the purpose of this research. If SRs reported non-identical searches for GS and PubMed, the findings would be reviewing the ability of the original reviewers to translate their searches, which again was not the intention.
One of the most challenging aspects is the frequency with which Google changes its functionality without giving any prior notice to its users. In 2012, GS changed its advanced searching features and removed the ability to limit results to specific domains and disciplines (for instance medicine). In March 2013, GS reduced the maximum number of articles it can show per page from 100 to 20, and in June the tilde operator, that was very usable to search for synonyms (frequently used as a replacement for truncation), was removed from the regular search engine (google.com), at the moment it is still available in GS. These changes have a major impact on searching, and when users asked whether some of these features would be reinstated, Google said little . Even more threatening is the fact that since the end of 2011, GS has disappeared from the menu of regular Google, thus making it harder to find for those users who do not already know of its existence, although results from GS and a link to a search in GS ('Scholarly articles for…’) often appear in the search results of regular Google. Because of Google's tendency to shut down applications it considers less frequently used (like recently Google Reader in June 2013), this might be a threat for the continuity of GS. And if GS were to be shut down, this would be a major threat to the replicability of the methodology of the SRs that were performed with a GS search.
Many published SRs report on the total number of hits in the databases they have used. However, the ratio between the number of hits we retrieved in PubMed and the total number of hits that were reported by the original authors varied a lot. In one case the reported total was 108 times higher than the number we found in PubMed, while for many other SRs the ratio between numbers reported and what was retrieved was equal to one. Some authors opted to report the total number of citations found in GS, while others took into account the first 1,000, or just reported the number they felt was necessary to view. Still, others ignored number of hits in GS in their reporting and counted only relevant, unique hits. Consequently, the resulting numbers in our searches were at odds with what the original authors reported in their published reviews. It is recommended that authors of SRs that use GS as a primary source report only those hits that were actually reviewed from GS.
Search reproducibility was low due to inaccurate or incomplete reporting of search strategies. Many papers that were examined referred to their search strategies by listing the keywords and Boolean operators used in an illogical order . Even in cases where searches were explicitly stated, the number of hits did not match the number of hits retrieved using the exact limiters and search parameters. To ensure transparency and reproducibility, authors of SRs should take care to follow the guidelines in the PRISMA Statement  for reporting search strategies. This states that the number of studies screened should be stated and not the number a database claims to have found.
Importing all references into reference management software is now a standard feature in bibliographic databases such as PubMed and Embase. GS does not offer such a feature. With Zotero, all results from one page can be imported from GS. However, recently the maximum number of hits shown per page changed from 100 to 20, making downloading the full set of hits more time consuming. When the authors of this article used Zotero to import the contents of a single search into Endnote, after downloading 200 references a 'Captcha’ was shown, as Google had detected that our 'computer or network may be sending automated queries’. GS seemed disinclined to provide the flexibility required to properly search the literature for the SRs.
We observed that many items were found because GS indexes content beyond the abstract into the full-text of articles, including their references. Excerpts often show a part of the article containing the reference list with the searched words found in titles of referred articles. When searching for included references in the first 1,000, numerous false hits were encountered showing the included articles in the reference list of other retrieved articles. GS seems to perform citation tracking for articles by using search words in the title. This accounts for much of the extra hits (or 'noise’) in GS’ results.
One limitation in this research (as in all retrospective research involving GS) is one can never be certain of GS’ coverage at any specific point in time, which is a serious problem for searchers. We performed searches several years after the original reviews were performed (average search date was September 2010 but ranged from Jan 2007 to October 2012, while we searched GS in May/October 2013). Although we limited GS searches to the publication year of the original search date, the results will probably differ from those retrieved by the original authors. Not surprisingly, the number of hits retrieved during our searching was often at odds with what was originally reported. Since replicable searching is essential in performing research, the inability to reproduce searches in GS severely limits its value to researchers. Bibliographic databases such as PubMed offer additional database management dates next to publication dates. These databases keep track of content, and note when changes are made. In PubMed using the field mesh date ([mhda]), one can be rather certain what a search result would have been on a given date. GS lacks these management dates, and only offers publication dates. Because of the absence of a clear date restriction feature, replicability, the search process (which is crucial to SR searching) is very is rendered far more problematic than would be the case in curated databases such as Medline and Embase.
We limited our searches to the publication dates of the last search date. However, this reduced the reported hits substantially. Even when a theoretical limit to publication years 1800 to 2099 was applied, the number of hits dropped. In simple queries, this seemed to affect only the number of hits, but not the resulting references. The first 1,000 articles remain largely the same, making this a good replacement for date limits in bibliographic databases. However, on the improved search strategies that were more complex, the number of hits reported often dropped a factor ten, even with the theoretical limit to all publications dates. Resulting references differed immensely, and the first 1,000 hits hardly contained any of the included references. Therefore, for the improved searches, publication date limits were not used. This is a newly identified problem when using GS for SR searching.
Though it was not our intention to judge the quality of the searches of the SRs, we believe that the quality of the searches was poor as they often only combined a few words with hardly any synonyms. This is of course also due to the selection process: if an identical search is used in GS, none of the more sophisticated tools of PubMed (for instance MeSH terms, field codes and truncation) could have been used.
Comparison of Google Scholar and PubMed in systematic review searching
Shows up to 1,000 results
Shows all results
Searches the full text of the article, and words on the webpage
Searches only bibliographic data and controlled vocabulary (MeSH terms)
No controlled vocabulary available
Controlled vocabulary (MeSH terms) added by skilled indexers and searchable (including 'explode’)
Searches the broad aspect of science (filters limiting results to medical articles were removed)
Only contains articles on medical topics
No search history available (unable to compare or combine record sets)
Detailed search history available, flexibility in combining record sets to create complicated search strategies
Search queries limited to 256 characters
No limits on the length of search queries
No truncation allowed. Tilde can be used to search for variants, but cannot be used in OR relationship with other words. GS is said to search for word variants, but this is very rare and the mechanism is unclear.
Possibly automatic searching for synonyms (details unclear)
Automatic Term Mapping (details available)
Field names only for title (complete query) and author names
Field names for many fields can be assigned per synonym
No advanced limits (for publication type, human studies and so on)
Multiple advanced limits in the database itself, or available from third parties
Cannot accurately limit to search dates (no controlled updates)
Different date fields available to limit searches to results before a certain date
Cannot download results in bulk to reference management software
Multiple options to download the complete results set to reference management software
Proximity searching only with exact order and exact number of connecting words
No proximity search possible
Though we wanted to improve the searches that returned the least included references, we could not draw conclusions about the effects of our improvements. We do not know, for example, if the authors missed important references in their original searches. We assume that within the extra hits retrieved, extra relevant studies might have been found. Because GS was unable to retrieve all articles found by the authors, its results cannot be considered complete.
As we've shown, the coverage and precision of GS are acceptable. Although coverage is not 100%, for many investigators 98% might suffice for simple literature or narrative reviews of a topic. The overall precision of GS (using the total reported number of hits as denominator) is rather low, but the practical precision, calculated by the number of relevant hits in the first 1,000, with 1,000 as denominator is theoretically acceptable for SR searching, and highly dependent on the number of included references.
Our a priori question was 'Is GS’s recall sufficient to be used on its own in systematic review (SR) searching?’ Overall retrieval in GS is 72%, which is too low for it to be used as a single database for the SR. PubMed fared similarly at 68%. The creation of better searches in GS proved to be a constant challenge. A high of 66% recall was achieved for the five searches that initially missed the most references from the SRs. In PubMed, our improved searches reached 85% recall. Neither database was sufficient on its own to find all articles from previously-published SRs.
Researchers from other disciplines might find results in GS that are 'good enough’. Similarly, medical professionals might find using PubMed (or another bibliographic database) on its own is good enough for day-to-day searching. Some may even prefer GS for initial searching to find 'a few good articles’ due to its excellent relevance ranking.
However, SRs require a complete view of all existing literature in a given area. This can only be achieved by performing exhaustive searches of relevant databases and websites in consultation with a trained information specialist. These important searches that support the SR methodology must be repeatable, verifiable and accountable, which poses a problem with GS.
There is therefore no reason why GS should be viewed as more suitable for performing SR searches than PubMed (or any other specialized database). We hope that our research will inform future authors and guide their use of GS. Authors of future SRs should continue to use GS but in concert with multiple other databases, not as a replacement of other databases.
WB is an information specialist at Erasmus MC, Rotterdam University Academic Hospital. He holds a BSc in biology and information science. He performs exhaustive searches for more than 200 systematic reviews a year.
DG is the UBC Biomedical Branch Librarian at Vancouver General Hospital in Canada. He holds MLS and MEd degrees.
BK is an information specialist at Utrecht University Library, working at the University Medical Center Utrecht. She holds a PhD in Neurobiology.
PA is Emerging Technologies Librarian at the University of Michigan. She holds an MLIS and a BSc in psychology and music.
The authors thank Jacqueline Limpens (Information Specialist, AMC Amsterdam) for her insightful and critical comments on earlier drafts of the article. She insisted we should try to optimize the original, often suboptimal searches in both databases for a thorough comparison, which added real scientific value to our article.
- Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P: Should meta-analysts search Embase in addition to Medline?. J Clin Epidemiol. 2003, 56: 943-955. 10.1016/S0895-4356(03)00110-0.View ArticlePubMed
- Watson RJ, Richardson PH: Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of Embase, Medline and PsycINFO bibliographic databases. Br J Med Psychol. 1999, 72 (Pt 4): 535-542.View ArticlePubMed
- Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA: Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005, 58: 867-873. 10.1016/j.jclinepi.2005.03.004.View ArticlePubMed
- Topfer LA, Parada A, Menon D, Noorani H, Perras C, Serra-Prat M: Comparison of literature searches on quality and costs for health technology assessment using the MEDLINE and EMBASE databases. Int J Technol Assess Health Care. 1999, 15: 297-303.PubMed
- Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L: Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Med Res Methodol. 2005, 5: 24-10.1186/1471-2288-5-24.PubMed CentralView ArticlePubMed
- Betran AP, Say L, Gulmezoglu AM, Allen T, Hampson L: Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005, 5: 6-10.1186/1471-2288-5-6.PubMed CentralView ArticlePubMed
- Moher D, Liberati A, Tetzlaff J, Altman DG: The Prisma Group: Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA Statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.PubMed CentralView ArticlePubMed
- McGowan J, Sampson M: Systematic reviews need systematic searchers. J Med Libr Assoc. 2005, 93: 74-80.PubMed CentralPubMed
- Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011,http://www.cochrane-handbook.org,
- Giustini D: How Google is changing medicine. BMJ. 2005, 331: 1487-1488. 10.1136/bmj.331.7531.1487.PubMed CentralView ArticlePubMed
- Chen X: Google Scholar's dramatic coverage improvement five years after debut. Ser Rev. 2010, 36: 221-226. 10.1016/j.serrev.2010.08.002.
- Harzing AW: A longitudinal study of Google Scholar coverage between 2012 and. Scientometrics. 2013, 2013: 1-11.
- Shultz M: Comparing test searches in PubMed and Google Scholar. J Med Libr Assoc. 2007, 95: 442-445. 10.3163/1536-5050.95.4.442.PubMed CentralView ArticlePubMed
- Falagas ME, Pitsouni EI, Malietzis GA, Pappas G: Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008, 22: 338-342.View ArticlePubMed
- Anders ME, Evans DP: Comparison of PubMed and Google Scholar literature searches. Respir Care. 2010, 55: 578-583.PubMed
- Nourbakhsh E, Nugent R, Wang H, Cevik C, Nugent K: Medical literature searches: a comparison of PubMed and Google Scholar. Health Info Libr J. 2012, 29: 214-222. 10.1111/j.1471-1842.2012.00992.x.View ArticlePubMed
- Walters WH: Google scholar search performance: comparative recall and precision. Portal. 2009, 9: 5-24.View Article
- Shariff SZ, Bejaimal SA, Sontrop JM, Iansavichus AV, Haynes RB, Weir MA, Garg AX: Retrieving clinical evidence: a comparison of PubMed and Google Scholar for quick clinical searches. J Med Internet Res. 2013, 15: e164-10.2196/jmir.2624.PubMed CentralView ArticlePubMed
- Gehanno J-F, Rollin L, Darmoni S: Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Med Inform Decis Mak. 2013, 13: 7-10.1186/1472-6947-13-7.PubMed CentralView ArticlePubMed
- Giustini D, Kamel Boulos MN: Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. 2013, 5: 214-PubMed CentralView ArticlePubMed
- Boeker M, Vach W, Motschall E: Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Meth. 2013, 13: 131-10.1186/1471-2288-13-131.View Article
- Freeman MK, Lauderdale SA, Kendrach MG, Woolley TW: Google Scholar versus PubMed in locating primary literature to answer drug-related questions. Ann Pharmacother. 2009, 43: 478-484. 10.1345/aph.1L223.View ArticlePubMed
- Haase A, Follmann M, Skipka G, Kirchner H: Developing search strategies for clinical practice guidelines in SUMSearch and Google Scholar and assessing their retrieval performance. BMC Med Res Methodol. 2007, 7: 28-10.1186/1471-2288-7-28.PubMed CentralView ArticlePubMed
- Henderson J: Google Scholar: a source for clinicians?. Can Med Assoc J. 2005, 172: 1549-1550. 10.1503/cmaj.050404.View Article
- Mastrangelo G, Fadda E, Rossi CR, Zamprogno E, Buja A, Cegolon L: Literature search on risk factors for sarcoma: PubMed and Google Scholar may be complementary sources. BMC Res Notes. 2010, 3: 131-10.1186/1756-0500-3-131.PubMed CentralView ArticlePubMed
- Tober M: PubMed, ScienceDirect, Scopus or Google Scholar - Which is the best search engine for an effective literature research in laser medicine?. Med Laser Appl. 2011, 26: 139-144. 10.1016/j.mla.2011.05.006.View Article
- Kennedy CM, Powell J, Payne TH, Ainsworth J, Boyd A, Buchan I: Active assistance technology for health-related behavior change: an interdisciplinary review. J Med Internet Res. 2012, 14: e80-10.2196/jmir.1893.PubMed CentralView ArticlePubMed
- Sampson M, Tetzlaff J, Urquhart C: Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Meth. 2011, 2: 119-125. 10.1002/jrsm.42.View Article
- Google Scholar Removes Search By Discipline. [http://www.seroundtable.com/google-scholar-discipline-16631.html]
- Golder S, Loke Y, McIntosh HM: Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008, 61: 440-448. 10.1016/j.jclinepi.2007.06.005.View ArticlePubMed
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.