Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?
© The Author(s). 2016
Received: 20 May 2016
Accepted: 1 November 2016
Published: 15 November 2016
Websites and online resources outside academic bibliographic databases can be significant sources for identifying literature, though there are challenges in searching and managing the results. These are pertinent to systematic reviews that are underpinned by principles of transparency, accountability and reproducibility. We consider how the conduct of searching these resources can be compatible with the principles of a systematic search. We present an approach to address some of the challenges. This is particularly relevant when websites are relied upon to identify important literature for a review. We recommend considering the process as three stages and having a considered rationale and sufficient recordkeeping at each stage that balances transparency with practicality of purpose. Advances in technology and recommendations for website providers are briefly discussed.
KeywordsSystematic reviews Information retrieval Website searching Online searching Information management
Many systematic reviews use topic-specific bibliographic databases to identify literature in a ‘systematic’ way. The functionality of these databases facilitates highly structured Boolean searching, automated recording of search history and bulk exporting of results. These functions support transparency, accountability and reproducibility of the search process, in line with accepted principles of literature searches for systematic reviews [1, 2]. However, literature is often sought outside of bibliographic databases, regardless of subject discipline or methodological focus of the review. Approaches might involve searching websites, search engines or online repositories and typically require searching and browsing (reading and navigating) techniques that differ from approaches to searching bibliographic databases. In comparison with bibliographic databases, there are greater challenges in deciding which websites and online resources to use, running complex searches, exporting search results and documenting the process. Problems encountered when searching websites with limited search functionality include large search outputs, empirical research hidden on websites within a wealth of other material and lack of abstracts . Where websites are relied upon to identify important literature for a review, it raises the issue of how the search is transparent, accountable and reproducible.
Our focus is on websites and online resources outside academic bibliographic databases. We use the term ‘websites’ in a broad sense to refer to online resources that lack the functionality to carry out complex Boolean searches, or export results, or do not readily provide a search history. Such resources vary widely in terms of appearance, functionality and content. They include websites of organisations, institutional repositories, research registers, online library catalogues and internet search engines. The value and rationale for utilising these resources varies between reviews and within review teams. Other complementary searching approaches include asking key contacts and authors, hand-searching journals, cited-reference searching and checking references.
We previously observed that relevant literature for low- and middle-income countries, such as working and policy papers, is often not included in databases, and is located from organisational websites, contacting authors or internet search engines . For some systematic reviews undertaken at the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), over a quarter of relevant citations were found from websites and internet search engines [4–13]. This finding is based on data from eight systematic reviews, of which four concern interventions in international development [4, 7, 12, 13], and four concern people’s views to inform to UK public health policy initiatives [5, 6, 8, 9].
While there is established guidance on conducting systematic searches of bibliographic databases, it is less clear how to approach searching websites for systematic reviews. We briefly describe our approach in Brunton et al.  and expand on this approach here. There is some specific guidance on web-searching for systematic reviews published by the Centre for Environmental Evidence , with emphasis on using search engines. Other related work is framed around searching for ‘grey literature’, where the aim is to seek out relevant literature that is not published in academic journals. Haddaway and Bayliss  consider grey literature in two forms: unpublished academic research and research that is generated by practitioners. They present different scenarios for undertaking searching for grey literature and suggest resources for each scenario. There are case studies demonstrating approaches in undertaking grey literature searching within public health-related topics published by Godin et al. , Mahood et al.  and McGrath et al. . Eysenbach et al.  provide an approach to internet searching for unpublished clinical trials. There are published studies on searching specific resources systematically, for example, Google Scholar [21–24] and trials registers . Outside of the systematic review literature, Blakeman  outlines challenges and approaches for searching Google effectively and discusses other online resources and tools for retrieving research.
The aim of this discussion is to: consider the challenges of searching websites and online resources outside academic bibliographic databases; to present an approach for conducting website searching for a systematic review; and consider how identifying literature from websites can be systematic in terms of being transparent, accountable, and reproducible. All the authors of this discussion have undertaken searching for systematic reviews across the fields of health systems and social care, public health, education, social policy and international development. Our approach draws on our experience from conducting systematic reviews, and supporting other review teams to undertake systematic reviews over many years and is informed by discussions within our research centre. We suggest website searching should not be considered only in relation to ‘grey literature searching’ because it can be used as a strategy to identify journal articles not identified from traditional bibliographic database searches. It might also be used to discover journal citations missed by a database search strategy, to compensate for poor access to subscription databases and for journals that are not indexed within any of the databases searched. We propose a systematic approach to the design and conduct of website searching and a method of recordkeeping. It is not our intention to describe methods for using specific resources. Neither is it to encompass reporting the search in a written published report, which is an area for separate consideration, for example, Briscoe  explored the reporting of how websites and search engines were searched in health technology assessments. We reflect on our approach in light of other published works, the potential implications of new technologies and make recommendations for website providers. We hope to promote further discussion of methods in literature searching for systematic reviews and other types of evidence syntheses.
How can we be systematic?
Three stages within systematic searches and challenges for website searches
Challenges in undertaking this step for websites and online resources
Planning the search
• To have a rationale for searching methods, based on the purpose of the search
• Planning where to search, who is undertaking the search and the timeframe of the review
• Discovery of suitable websites to use
• Deciding how representative the range of websites need to be in relation to the scope of the review and time available to search
Executing the search
• To utilise each resource in a consistent way, and in a way that is appropriate for each individual resource
• Planning how to search when each source is structured differently and may differ in terms of focus and content
• Using individual approaches for each website
• Searching resources where the functionality for searches consisting of multiple words and Boolean searching is often limited
Screening and information management
• To assess literature for relevance (screening or sifting)
• To quantify how many items of literature is processed
• To report on methods
• Which literature to collect and how much screening to carry out
• Limited functionality to transfer results to citation management tools
• The level of detail needed for recordkeeping for preliminary screening at source
Planning the search
Planning the search involves having a rationale to justify and inform decisions on where to search. It also considers who is undertaking the search and the timeframe and resources available for the review. The role and purpose of website searching compared with other methods of identifying literature informs these decisions. One challenge is knowing about the most appropriate websites to search. Unlike selecting bibliographic databases, which often cover broad topic areas and specific disciplines, identifying appropriate websites is more dependent on the precise nature of the research question and knowledge and accessibility of the websites available. There is a vast range of options that vary in scope, functionality for searching and browsing and volume of content. The choice of websites should reflect those most suitable to the review, and includes deciding how representative of the topic of investigation it needs to be. There is potential for introducing unintentional bias; for example, a review covering low- and middle-income countries worldwide involves searching a combination of websites that span relevant geographical areas, and is not limited to one geographical region. Another bias could be introduced by focussing only on sources relating to a particular stakeholder group, age group, setting, or study design without appropriate reasons. It can take considerable time to search individual websites, particularly those of individual organisations, or those that contain long publication lists. A risk is that the process will not yield any unique or relevant records compared with other searching techniques, and time is spent looking at references discovered elsewhere.
Examples of choosing websites for different reviews
Key purpose of website search
Types of websites, online resources and depositories
Access to economic assets for women in low- and lower-middle-income countries 
Discover relevant research missed or not indexed in international or regional databases
Over 35 sites consisting of government and research-active non-governmental organisations, academic research centres and funders, relating to economics, microfinance, international development, or regional development banks
Adult cooking skills programmes 
Discover unpublished evaluations of cooking skills programmes in the UK
Generic search engine, library catalogues, and 25 websites of UK public health and community organisations, research centres and government departments
Depression, anxiety, pain and quality of life in people living with chronic hepatitis C 
Discover research identified by advocacy organisations and health research potentially missed by database searches
Websites of hepatitis C advocacy groups in mainly in the UK and some resources to containing healthcare research in general
Realist synthesis of school accountability in low- and middle-income countries 
Undertake purposive searching for advisory group engagement and scoping exercise stages prior to bibliographic database searching
Specialist databases, search engine, 20 websites of international development agencies and organisations
Exercise interventions and patient beliefs for people with chronic hip and knee pain 
Discover literature on people’s experiences, largely unpublished in journals
Range of website resources covering: arthritis groups in UK, Australasia, and North America, ageing care registries, patient experience resource, grey literature resources, generic search engine, social science research
It is important to consider which review team member will undertake the searches and ensure they have sufficient understanding of the type of information that is being sought from the literature search, as well as skills in locating and managing literature found from the websites. If a review team has a policy of screening publications for eligibility by two people, they need to decide whether to extend this for website searching, and their rationale for doing so; for example, if the aim is for consistency or to help ensure relevant items are not missed. Given the potential variation by individuals in searching websites, it seems easier to operationalise the latter, without striving to match the exact process used by each person.
Another aspect to consider is the time-point of website searching alongside the rest of the systematic review. For example, if the website search is undertaken by a reviewer who has already screened literature against the eligibility criteria of the review, they would have a clear idea of the literature sought from websites. Hammerstrom et al.  suggest that completing web searches towards the end of the search phase of a review ensures picking up the most current information. On the other hand, searches undertaken at an early stage of the review may inform any bibliographic database searching or other searching techniques that might be planned later. Overall, we think that the searching could take place at any point, depending on what is needed for an individual review.
Executing the search
The objective of searching should be to utilise each resource in a consistent way and in a way that is appropriate for each individual resource. This poses particular challenges for websites because each resource is structured differently and may differ in terms of focus, content and functionality for searching and browsing. It is difficult to judge how to search with confidence that items of interest have not been missed. It is impossible, and potentially unhelpful, to treat each resource in the same way. Each website requires different techniques, for example, browsing relevant web pages, searching using a generic search function, navigating headings within webpages or scanning lists of references. It is likely that more than one approach is needed for each website, and time is needed to develop knowledge and skills to utilise individual resources.
From our experience, the process of recording how a website is searched helps in considering the search approach. The act of recording which navigation headings are browsed and which search terms are used helps the searcher to reflect on their choice and rationale of approach and may prompt useful iteration of searches. It aids structuring a search for each website, encourages a greater level of care to be taken when searching and enables comparison across different websites. Recordkeeping may help in using a consistent approach for similar resources, while at the same time giving flexibility to search each resource differently, as needed. As well as facilitating searching, such recordkeeping provides a degree of transparency and aids accountability and reproducibility for internal documentation. It also enables knowledge and skills gained from using particular websites to inform future searches, for example, if revisiting the same websites at a later date to search using different terms or to update the original search.
Example of recordkeeping for executing the search
Name of resource
Date of last access (if different)
Pathway followed, e.g. browsed headings/searched site/database within website (use separate lines for the different types of searches)
Used predefined keywords ‘Y’
Alzheimer’s Society Research https://www.alzheimers.org.uk/research
Browsed ongoing and completed research on the topic of ‘Towards better care for people with dementia’
browsed ‘living with dementia’ research articles
Searched dementia catalogue—subject: hospital admissions, hospital discharges
Searched dementia catalogue-free text: transition, transitions or transfer
Searched descriptor field: Hospitals AND (Psychiatry OR Psychiatric Rehabilitation) AND 1999–2015
We consider the process of searching and browsing as iterative, as the content within each website might prompt using different search terms, or browsing other parts of a website. Some searches may incorporate a full-text search of a document, which could require adapting search terms to increase the relevance of results. It is important to be aware of user bias in terminology and to use the headings or index terms set by the website provider or consider browsing records to complement using search terms. Establishing a pool of terms to draw on or reflecting on searches for similar websites can help.
Actual searching and browsing methods are likely to differ from review to review. For example, some review teams may adopt a uniform method to apply to specific groups of resources. On the other hand, it may be more appropriate to adopt varying approaches that are individual to a website, but consistent with the overall premise of finding relevant research that meet the criteria for a review. Depending on the content of the resource and how it is structured, it may be appropriate to browse references, rather than run searches on keywords, or use a combination of approaches. Searching some registers or websites focussed on an area of relevance provides an opportunity to search more broadly than is practical with a bibliographic database, as the number of results is likely to be much smaller. For example, for a series of reviews relating to the transition between inpatient mental health and community and care home settings, we found separate searches with the terms ‘hospital’ or ‘psychiatric’ specific enough to identify a small number of records on some websites, but too generic to use for others.
Schucan-Bird and Tripney  describe separate approaches used for searching websites of organisations, subject specialists, research funding bodies and Google for a large systematic literature search. They adopted a general approach for websites of organisations, which involved browsing all items listed under one section where the publications numbered less than 100, and used a search function drawing on a pool of search terms where they found over 100 publications were listed. In contrast, Godin et al.  describe a different approach where they searched a website database or used the search function and hand-searched where these functions were not available. Mahood et al.  describe an approach where they compiled a pool of search terms, customised the search and used controlled vocabulary wherever possible. In several databases, a simple strategy of two key terms that could be truncated was used in addition to a full or modified longer set of search terms to ensure a comprehensive search. For online repositories, they used their simple strategy in various fields (title, abstract, where available), with result yields varying from two to over 500 references.
Evaluations on utilising specific resources can guide practice. Glanville et al.  studied search approaches for two clinical trials registries, ICTRP and Clinicaltrials.gov, and found single-concept searches in the basic interfaces to be the most reliable. Haddaway et al.  investigated approaches for searching Google Scholar, which only displays the first 1000 references of a search; they found that title searches enabled discovery of more grey literature (conference proceedings, theses, reports) than full-text searches. They also found that these types of publications occur later in the ranked list of results than academic journal papers.
Screening and information management
Once a resource is searched, it must be decided which results should be saved for screening (or sifting) for relevance against the eligibility criteria for a review. This stage also involves recordkeeping to quantify how many literature citations have been processed and the methods used for selecting potentially relevant literature.
There is often no function to export results automatically into citation management tools, so challenges include deciding which literature to keep from each resource, how much screening to carry out within each resource and the level of detail for recordkeeping for preliminary screening of results. Manually transferring all the results is possible, but usually some boundaries need to be set on what is transferred; otherwise, it is an inefficient exercise to collect all references, regardless of their relevance, so they can be screened in a systematic way. Furthermore, transferring all results could promote bias in only undertaking highly focused searches, so that the results manageable, and hinder expansive browsing and iterative searches. We expect that preliminary screening within the website is necessary so that only items that are relevant are saved for further examination. In this way, it can be helpful to consider searching and screening as continuous and iterative. Conceptually searching and screening are on a continuum, as they are both aimed at narrowing a collection of research into those most relevant to answer a review question. However, it is important to consider how much transparency is needed in describing the screening of studies at source.
In our opinion, an efficient way to view screening on websites is to seek out only the items that are of likely relevance to a review and record the number collected for formal screening against eligibility criteria, rather than recording the decision made about the relevance of every citation that is encountered. This approach focuses on describing the literature found. It is possible to have some transparency in assessing the results. For example, indicating whether the literature was assessed for relevance on the basis of the title alone, title and abstract or full text. Where partial lists are browsed, an indication of how that list was organised should be provided (for example, scanning the first 100 items by relevance). Reporting and screening may vary between each resource, as results might be displayed in different ways (for example, a list, a selected quantity displayed by relevance, or chronologically). Where results are scanned by relevance, it depends on user-judgement of how many is appropriate to scan for that particular source.
Example of recordkeeping for initial screening and information management
Name of resource
State Y if automated exporting in RIS/XML
If no automated exporting available:
No. of promising documents
Approach to screening, e.g. title, then abstract/full text OR first 100 ranked by relevance
Alzheimer’s Society Research https://www.alzheimers.org.uk/research
69, 23, 43 items from the searches
Screening at source may be particularly time-consuming depending on the type of literature or if the results contain a lot of relevant records that have already been identified from other searches. Mahood et al.  observe that with non-journal literature, it is sometimes difficult to judge relevance based on titles and abstracts due to missing citation information or abstracts. They also observe that duplicate references from Google and Google Scholar can be difficult to recognise due to different citation formats and missing citation information. One approach we have used with Google Scholar is to individually export the results into a citation management tool and duplicate check these against records collected elsewhere, leaving a smaller number of citations to examine. Godin et al.  use a bookmarking system within their web browser to avoid identifying the same record twice as the URL of previously bookmarked pages are starred.
Where an internet search engine yielding vast numbers of results in a ranked order, one must decide how many results to screen. This may be informed by the results returned and then screening to saturation (for example, scanning until no more relevant items are identified on a page or on a following page). Based upon a utility analysis of Google Scholar for seven reviews in environmental science, Haddaway et al.  recommend looking at the first 300 results in Google Scholar for academic literature, screening well beyond this to find relevant grey literature and they advocate the use of tools to gain a snapshot of the first 1000 results.
Advances in technology
Advances in technology are likely to increase options for automating website searching, data and document retrieval and recordkeeping in the future. Automated logging tools that store search history, browsing patterns and saving of content have the potential to assist the entire process. For example, the Open Source Internet Research Tool (http://osirtbrowser.com/) provides screen captures, a log of the locations and time a webpage was browsed, fields for the user to record their own notes and also facilitates file management. Other tools may be helpful when searching websites. For example, bookmark management and screen-clipping tools can be used to save and organise information. Web-scraping tools, such as import.io, which extract data from websites, are becoming accessible to users without programming expertise, and may be a useful application for exploring and managing the content examined from some websites.
These tools have the potential appeal for reducing manual recordkeeping. However, the elements logged need to be meaningful, and their use could possibly affect the processes of reflection and iteration encountered in searching. Perhaps these processes illustrate two contrasting approaches: (a) targeted hand-searching such as browsing and manual assessment and retrieval of relevant items and (b) capturing large quantities of studies, including many irrelevant studies, using automated tools and filtering for relevance using text-mining technologies. We expect a hybrid of both approaches would co-exist. Overall, their performance in comparison with manual browsing, searching and document retrieval from individual websites needs to be considered.
Transparency, accountability and reproducibility
Recordkeeping provides transparency, accountability and reproducibility of the process to varying extents. Transparency is achieved by recording brief information on a resource searched, how and when it was searched, the approach to screening for relevant literature and the number of relevant items saved for further assessment. For many systematic reviews, we have captured the information as an internal record of what was done, though it is potentially available to a wider audience. Publishing fuller details of the search process may be appropriate where there is more reliance on website sources than on traditional bibliographic databases. Accountability is achieved to some extent by having a rationale for the resources searched and having a record of how the search was carried out. This rationale is influenced by several factors, such as the knowledge and skills of the searcher, and time and resource constraints under which a review takes place. A related aspect is that recordkeeping helps searchers consider how they are searching, thereby improving the quality of the search undertaken. These factors are largely hidden from an independent reader of a systematic review, but influence the conduct of searches. Reproducibility of searches is achieved to a limited extent whereby the general approach taken could potentially be replicated. However, clear, systematic and replicable approaches to searching might not lead to replicable results, as observed by Adams et al. . Within each resource, reproducibility is limited as the content of the resources, search functionality and underlying search mechanisms are not static. Furthermore, how they are searched depends on the different perspectives and skills of a user. Limitations for reproducibility do not outweigh the advantage of searching these resources, and we suggest a greater emphasis on transparency and accountability is more appropriate.
Recommendations for website providers
• Consider how users can discover the most relevant content
• Provide instructions or labels for locating research
• Separate empirical research from opinion pieces and guidance tools
• Make all search results available
• Inform users how results are displayed (e.g. relevancy ranked, date published, date added)
• Provide functionality to export citations into citation management tools
Thank you to our colleagues at the EPPI-Centre involved in discussions of this work, particularly Ginny Brunton, David Gough, Alison O' Mara-Eves, James Thomas and Jan Tripney.
An early version of this work was presented as a poster: Stansfield C, Dickson K, Tripney J, Bangpan M, Stewart R, Vigurs C, Hauari H, Oliver S (2013) Finding research on websites - experiences and solutions from the field of international development (Poster). 21st Cochrane Colloquium, Quebec, 19-23 Sept.
This work received no funding.
Availability of supporting data and materials
All authors have participated in the conception, design, drafting and approval of this manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Hammerstrøm K, Wade A, Jørgensen AMK. Searching for studies: a guide to information retrieval for Campbell Systematic Reviews Campbell Systematic Reviews 2010; Supplement 1 DOI: https://doi.org/10.4073/csrs.2010.1.
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. http://handbook.cochrane.org/. Accessed 22 Mar 2016.
- Oliver S, Bangpan M, Stansfield C, Stewart R. Capacity for conducting systematic reviews in low- and middle-income countries: a rapid appraisal. Health Res Policy Syst. 2015;13:23. doi:https://doi.org/10.1186/s12961-015-0012-0.View ArticlePubMedPubMed CentralGoogle Scholar
- Birdthistle I, Dickson K, Freeman M, Javidi L. What impact does the provision of separate toilets for girls at schools have on their primary and secondary school enrolment, attendance and completion? A systematic review of the evidence. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011.Google Scholar
- Brunton G, Oliver S, Oliver K, Lorenc T. A synthesis of research addressing children's, young people's and parents' views of walking and cycling for transport. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=942.Google Scholar
- Brunton G, Wiggins M, Oakley A. Becoming a mother: a research synthesis of women's experiences of first-time motherhood. EPPI Centre, Social Science Research Unit, Institute of Education, University of London. 2011 http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3295.
- Dickson K, Bangpan M. Providing access to economic assets for girls and young women in low-and-lower middle income countries: a systematic review of the evidence. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2012. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3359. ISBN 978-1-907345-38-8.Google Scholar
- Rees R, Oliver K, Woodman J, Thomas J. The views of young children in the UK about obesity, body size, shape and weight: a systematic review. BMC Public Health. 2011;11:188.View ArticlePubMedPubMed CentralGoogle Scholar
- Rees R, Caird J, Dickson K, Vigurs C, Thomas J. ‘It’s on your conscience all the time’: a systematic review of qualitative studies examining views on obesity among young people aged 12–18 years in the UK. BMJ Open. 2014;4:e004404. doi:https://doi.org/10.1136/bmjopen-2013-004404.View ArticlePubMedPubMed CentralGoogle Scholar
- Stansfield C, Brunton G, Rees R. Search wide, dig deep: literature searching for qualitative research studies. An analysis of the publication formats and information sources used for four systematic reviews in public health. Res Synth Methods. 2014;5(2):142–51.View ArticlePubMedGoogle Scholar
- Stansfield C, Dickson K. Locating evidence for developing countries: a case study of three public health reviews. Poster presentation at the 19th Cochrane Colloquium; 2011 Oct 19-22; Madrid, Spain [abstract] Cochrane Database of Syst Rev. 2011;Suppl(CD000003), 51-2, CMR-16617.Google Scholar
- Stewart R, van Rooyen C, Dickson K, Majoro M, de Wet T. What is the impact of microfinance on poor people? A systematic review of evidence from sub-Saharan Africa. Technical report. London: EPPI-Centre, Social Science Research Unit, University of London; 2010.Google Scholar
- Tripney J, Schucan Bird K, Kwan I, Kavanagh J. The impact of post-abortion care family planning counselling and services in low-income countries: a systematic review of the evidence. Technical report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011.Google Scholar
- Brunton G, Stansfield C, Caird J, Thomas J. Chapter: 6: finding relevant studies. In: Gough, D. Oliver S, Thomas J, editors. An introduction to systematic reviews. 2nd Edition. London: Sage Publications; 2017. In press.Google Scholar
- Collaboration for Environmental Evidence. Guidelines for systematic review and evidence synthesis in environmental management. Version 4.2. Environmental Evidence: Centre for Evidence-Based Conservation, Bangor University. 2013 www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf, Accessed 22 Mar 2016.
- Haddaway N, Bayliss H. Shades of grey: two forms of grey literature important for reviews in conservation. Biol Conserv. 2015;191:827–9.View ArticleGoogle Scholar
- Godin K, Stapleton J, Kirkpatrick SI, Hanning RM, Leatherdale ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015;4:138.View ArticlePubMedPubMed CentralGoogle Scholar
- Mahood Q, Eerd DV, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2013;5:221–34.View ArticlePubMedGoogle Scholar
- McGrath Y, Sumnall H, Edmonds K, McVeigh J, Bellis M. Review of grey literature on drug prevention among young people. National Institute for Health and Clinical Excellence. ISBN:1-84629-184-4. 2006.Google Scholar
- Eysenbach G, Tuische J, Diepgen TL. Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Med Inform Internet Med. 2001;26:203–18.View ArticlePubMedGoogle Scholar
- Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Meth. 2013;13:131.View ArticleGoogle Scholar
- Bramer W, Giustini D, Kramer B. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev. 2016;5:39.View ArticlePubMedPubMed CentralGoogle Scholar
- Bramer WM, Giustini D, Kramer BMR, Anderson PF. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2:115.View ArticlePubMedPubMed CentralGoogle Scholar
- Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015;10:e0138237.View ArticlePubMedPubMed CentralGoogle Scholar
- Glanville JM, Duffy S, McCool R, Varley D. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? J Med Libr Assoc. 2014;102(3):177–83.View ArticlePubMedPubMed CentralGoogle Scholar
- Blakeman K. Finding research information on the web: how to make the most of Google and other free search tools. Sci Prog. 2013;96(Pt 1):61–84.View ArticlePubMedGoogle Scholar
- Briscoe S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Res Notes. 2015;8:153.View ArticlePubMedPubMed CentralGoogle Scholar
- Schucan Bird K, Tripney J. Systematic literature searching in policy relevant, inter-disciplinary reviews: an example from culture and sport. Res Synth Methods. 2011;2(3):163–73.View ArticlePubMedGoogle Scholar
- Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.View ArticlePubMedGoogle Scholar
- Adams J, Hillier-Brown F, Moore H, Lake A, Araujo-Soares V, White M, Summerbell C. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Syst Rev. 2016;5:164.View ArticlePubMedPubMed CentralGoogle Scholar
- Rees R, Hinds K, Dickson K, O'Mara-Eves A, Thomas J. Communities that cook: a systematic review of the effectiveness and appropriateness of interventions to introduce adults to home cooking. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2012. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3322.Google Scholar
- Brunton G, Caird J, Sutcliffe K, Rees R, Stokes G, Oliver S, Stansfield C, Llewellyn A, Simmonds M, Thomas J. Depression, anxiety, pain and quality of life in people living with chronic hepatitis C: a systematic review and meta-analysis. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London; 2015.Google Scholar
- Spicer D, Ehren M, Bangpan M, Khatwa M (2014) Under what conditions do inspection, monitoring and assessment improve system efficiency, service delivery and learning outcomes for the poorest and most marginalised? A realist synthesis of school accountability in low- and middle-income countries. Protocol. London: EPPI-Centre, Social Science Research Centre, Institute of Education, University of London.Google Scholar
- Hurley M, Dickson K, Walsh N, Hauari H, Grant R, Cumming J, Oliver S. Exercise interventions and patient beliefs for people with chronic hip and knee pain: a mixed methods review (protocol). Cochrane Database of Systematic Reviews. 2013, Issue 12.Google Scholar