Continuous surveillance of a pregnancy clinical guideline: an early experience
Systematic Reviews volume 6, Article number: 143 (2017)
To date there is no consensus about the optimal strategy for keeping clinical guidelines (CGs) up-to-date. The aims of this study were (1) to develop a continuous surveillance and updating strategy for CGs and (2) to test the strategy in a specific CG.
The main steps were as follows: (1) recruiting members for the CG Updating Working Group, (2) mapping the CG, (3) identifying new evidence from the CG Updating Working Group, (4) designing and running restricted literature searches, (5) reviewing drugs and medical devices alerts, (6) screening and assessing the new evidence, (7) reviewing and, if necessary, modifying clinical questions and recommendations, and (8) updating the CG document.
The Pregnancy CG Updating Working Group consisted of 29 members, including clinicians, patients and caregivers, and clinical guideline methodology experts. We selected 69 clinical questions (123 recommendations) from the “Assistance during pregnancy” section.
For the first update cycle (32-month duration), 9710 references were identified. Of these, 318 were pertinent, 289 were relevant, and 55 were classified as potential key references. For the second and third update cycles (6-month duration each), 2160 and 2010 references were retrieved, respectively. The continuous surveillance and updating strategy has not yet been completely implemented.
Further resources are needed in updating the CG field, both for implementing updating strategies and for developing methodological research.
Clinical guidelines (CGs) are useful tools to help patients, health care providers, and policymakers make evidence-based decisions about health care. Consequently, they need to be updated in order to guarantee the validity of their recommendations.
Time of validity
Several studies have assessed the time of validity of CGs and their recommendations (defined as “time between the publication of a CG and the identification of new relevant evidence”) [1,2,3,4,5]. Data showed that recommendations quickly became outdated (about 20% of the recommendations were out of date within 3 years) . Based on this evidence, 40% (14/35) methodological handbooks typically recommend reviewing and updating guidelines within 2 to 3 years of their publication . Nevertheless, CG developers acknowledge that it is challenging to maintain these predetermined timeframes .
The updating of CGs is an iterative process that involves an explicit and systematic methodological approach for the identification and assessment of new evidence not included in the original CG [6, 8, 9]. If new relevant evidence is identified and it was considered to have an impact on the current CG, the CG should be modified, if necessary [6, 8, 9]. Moreover, the updating strategies provide an opportunity to improve the overall methodology and edition of the document (e.g. correction of mistakes or enhancement to the writing) [6, 8, 9].
To date, there is no real consensus on the optimal strategy for updating CGs [6, 7, 10]. Most of the available methodological research focuses on identifying new relevant evidence. Research suggests that pragmatic search strategies (with the aim of favouring precision over sensitivity) are efficient and feasible for retrieving new evidence that triggers a recommendation update [6, 11, 12].
Living clinical guidelines
To address the updating of CGs, the majority of CG developers support the concept of living CGs , generally defined as “prospective and active processes that use continuous surveillance and a rapid response to include new relevant evidence identified” . However, until now, no guidance has been developed to put this concept in practice , and a few empirical experiences were published [13,14,15,16,17]. CG developers considering the transition to living CGs will have to address challenges to operationalize the process .
In order to address some of the challenges related with living CGs, this study aimed to (1) develop a continuous surveillance and updating strategy for CGs and (2) test the strategy in a specific CG.
We conducted a cohort study of recommendations from the Assistance During Pregnancy and Puerperium CG included in the Spanish National Health System CG Programme .
The strategy was developed based on published methodological research and the experience of the CG Updating Working Group technical team [2, 6,7,8, 10, 11, 20, 21]. The processes included in the strategy were as follows: (1) recruitment of members for the CG Updating Working Group, (2) mapping of the CG, (3) identification of evidence from the CG Updating Working Group, (4) designing of restricted literature search strategy, (5) running of restricted literature searches, (6) reviewing alerts for drugs and medical devices, (7) development of reference database, (8) first reference screening, (9) second reference screening (assessment of new evidence impact), (10) development of a clinical questions database, (11) classification of clinical questions, (12) review and, if necessary, modification of clinical questions and recommendations, and (13) update of the CG manuscript (Table 1). Following, we provided a detailed description of the most complex processes: (1) identification of new evidence, (2) reference screening, and (3) classification of clinical questions.
Identification of new evidence
Three different strategies were used to identify new evidence: (1) a questionnaire sent to the CG Updating Working Group, (2) a restricted literature search strategy, and (3) a revision of available drugs and medical devices alerts.
The questionnaire sent to the CG Updating Working Group aimed to identify any new relevant evidence that could have an impact on the CG (questionnaire available from the authors upon request). The questionnaire covered the different areas of the CG including the scope, new potential aspects not included in the original version, or new relevant evidence assessing the effectiveness and safety of the interventions. The survey also included questions about other relevant factors such as changes in the relative importance of the outcomes, changes in the resource use and cost of the interventions, equity, acceptability, or feasibility issues that might have arisen since the publication of the CG. Information about on-going research studies was also sought in the survey.
Restrictive literature search strategies were developed for each clinical question in MEDLINE (PubMed) following a validated methodology described elsewhere . In summary, the minimum number of Medical Subject Headings (MeSH) terms and text words required from the original exhaustive search strategies were selected. The strategies were validated checking that all key references supporting the recommendations in the original CG were retrieved and were refined if needed. Once the strategies were validated, PubMed Clinical Queries filters were applied (www.ncbi.nlm.nih.gov/pubmed/clinical). Search strategies by topic were also developed using a specific filter to identify studies on how patients and other stakeholders value health outcomes and economic studies [22, 23].
Finally, drugs and medical devices alerts published by the Spanish Agency for Medicines and Health Products were reviewed (www.aemps.gob.es/en/home.htm).
The references were sequentially classified in the following:
Pertinent references: topic-related references that met the study design criteria
Relevant references: pertinent references that could be used when considering an update to a recommendation, but that would not necessarily trigger a potential update
Potential key references: relevant references that could potentially trigger an update
A specific questionnaire for each clinical question was used to identify relevant and key references (questionnaire available from the authors upon request). The questionnaire included the clinical question, the recommendations, and the references considered pertinent in the first screening to that clinical question. If the reference was considered relevant for that particular question by the reviewer, then it was deemed necessary to assess if the reference could potentially trigger an update (key reference). If it was the case, it was necessary to explicitly state which part of the question and/or recommendations was affected (population, intervention, comparator, outcomes, resource use and costs, equity, acceptability, feasibility, strength or direction of the recommendation).
Classification of clinical questions
Each clinical question was classified in one of the following categories:
Clinical question to be reviewed: question with potential key references or with alerts
Valid clinical question: question without potential key references or without alerts
New clinical question
Once the questions were classified, we planned to update them following a similar method used in the development of the original recommendations but taking into account the new evidence identified and the evidence used to develop the recommendations.
Conducting the 13 processes was considered a one update cycle. The first update cycle included new evidence since the last search date in the CG development process up to the first search date in the CG surveillance process; subsequent update cycles were scheduled every 6 months .
We performed a descriptive analysis of the data: literature search time periods, number of identified references, number of screened references, and number of classified references (pertinent, relevant, and key). We described narratively the steps achieved.
Clinical Guideline Updating Working Group
All members of the Pregnancy CG Development Working Group were initially contacted (20 members). However, since only five agreed to participate in this study, 30 additional candidates were contacted. The Pregnancy CG Updating Working Group finally consisted of 29 members: (1) clinical team: three medical specialists in gynaecology and obstetrics, three medical specialists in family and community medicine, and three midwives; (2) patients and caregivers team: three patients or patient representatives; and (3) technical team: 17 CG methodologists.
We identified 89 clinical questions and 201 recommendations in the Assistance During Pregnancy and Puerperium CG . We focused specifically on the “Assistance during pregnancy” section, which contained 69 clinical questions and 123 recommendations (36 strong recommendations, 49 weak recommendations, and 38 good clinical practice statements). We also retrieved the references used to support recommendations, original literature search strategies, evidence syntheses, and GRADE evidence profiles.
Continuous surveillance process
We contacted a total of 13 members of the Pregnancy CG Updating Working Group for the baseline survey and received 11 responses (84.6% response rate). We developed one search strategy per clinical question (a total of 62, as 7 clinical questions were clustered together), as well as topic searches for studies on patients’ values and preferences and for costs and resource use.
We identified 26 recommendations (26/123; 21.1%) related to drugs or dietary supplements. We consulted drugs and medical devices alerts from the Spanish Agency for Medicines and Health Products, searching by CG as topic.
For the first literature search cycle (32-month period), 9710 references were identified. Of these, the technical team classified 318 as pertinent, 289 as relevant, and 55 as potential key references (Table 2).
For the second and third literature search cycles (each a 6-month period), 2160 and 2010 references were retrieved, respectively (Table 2).
The surveillance process lasted 1 year, from November 2014, when the CG development institution was contacted for establishing the Pregnancy CG Updating Working Group, until November 2015, when the study was stopped early due to budgetary constraints.
The continuous surveillance and updating strategy has not yet been completely implemented. We have not assessed the results of the second and third cycles of the literature search or gauged the effect on recommendations of potential key references. As such, we have not reviewed and, if necessary, updated the CG recommendations.
We designed a step-by-step process for continuous surveillance and updating CGs. We were able to implement a continuous and restricted literature search strategy for the “Clinical Practice Guideline on Assistance during Pregnancy and Puerperium” for a 1-year period. In the first update cycle we identified 9710 references (318 pertinent, 289 relevant, and 55 potential key references). For the second and third update cycles 2160 and 2010 references were retrieved, respectively.
The continuous surveillance and updating strategy has not yet been completely implemented due to budgetary constraints.
Our results in the context of previous research
Only one previous study, published in 2003, assessed a continuous surveillance and updating strategy for CGs, specifically for cancer guidelines. This approach included a continuous and exhaustive literature search strategy, evaluation of the newly found evidence, review and updating of recommendations, and dissemination of the new evidence and modified recommendations among stakeholders. Similarly to our experience, the authors of this study highlighted the considerable resources required .
Other initiatives have ventured the implementation of new technologies to facilitate the CG updating process. One of them, called MAGIC (Making GRADE the Irresistible Choice) provides a publication platform where the main content of CGs can be disseminated. MAGIC also facilitates uploading modifications, including any potential updates, which would be available to users instantly . Similarly, “Kidney Disease: Improving Global Outcomes (KDIGO)” recently published a series of recommendations for a continuous, dynamic strategy for maintaining their CGs current. Their model heavily relies on the availability and processing of new evidence using integrated electronic platforms . Unfortunately, these new technologies have not yet been formally implemented and evaluated.
Strengths and limitations
We were able to retrieve, organise, and map the original documentation related to the development of the assessed CG, including the clinical questions, recommendations and references, original literature search strategies, evidence syntheses, and GRADE evidence profiles. We also adopted a systematic and continuous approach (every 6 months) to identify new evidence and to assess its impact on the CG recommendations. Lastly, we introduced evidence searches for patients’ values and preferences and for costs and resource use in the surveillance process.
However, our work is subject to some limitations. First of all, we have not been able to assess the impact of the new evidence on clinical questions and recommendations for either the second or third update cycles. In addition, we have not reviewed or modified clinical questions and recommendations based on the identified new evidence in any of the update cycles.
Second, we had difficulties assembling the Pregnancy CG Updating Working Group. The vast majority of the CG Developing Group did not take part in the implementation of the strategy. Hence, an almost new working group had to be set up for this purpose. On the other hand, some members of the Pregnancy CG Updating Working Group withdrew during the study, probably due to an excessive study-related workload (appraisal of a high volume of publications in the first update cycle, inadequate training related to the implementation of the strategy, and/or a lack of knowledge of the content of the original CG).
Third, the first surveillance cycle was quite resource-intensive (from the last search date in the original CG development process to the first search date in the CG surveillance process) and required the retrieval, mapping, and classification of the documentation generated during the development of the original CG. The process was optimised in the subsequent update cycles (second and third) given that (1) the process had already started and (2) the time between cycles (6 months) and, consequently, the volume of references (approx. 2000 references) were smaller.
Lastly, and related to the previous limitation, we did not have adequate funding to take on the management and development of the completely continuous surveillance and updating strategy we originally intended to implement. CG developers should consider using different surveillance and updating strategies to maintain their CGs up-to-date (a living strategy might not be suitable for all CGs). More research is needed to identify which CGs, topics, or areas could benefit from this or other approaches.
Implementing a continuous and restricted literature search process is a potentially feasible approach for the surveillance of new evidence. A continuous surveillance and updating strategy (as living CG) requires long-term substantial resources for its adoption. Further resources are needed in the updating CG field, both for implementing updating strategies and for developing methodological research.
Alderson LJ, Alderson P, Tan T. Median life span of a cohort of National Institute for Health and Care Excellence clinical guidelines was about 60 months. J Clin Epidemiol. 2014;67(1):52–5.
Martínez García L, Sanabria AJ, Garcia Alvarez E, Trujillo-Martin MM, Etxeandia-Ikobaltzeta I, Kotzeva A, et al. The validity of recommendations from clinical guidelines: a survival analysis. CMAJ. 2014;186(16):1211–9.
Neuman MD, Goldstein JN, Cirullo MA, et al. Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations. JAMA. 2014;311(20):2092–100.
Lyratzopoulos G, Barnes S, Stegenga H, Peden S, Campbell B. Updating clinical practice recommendations: is it worthwhile and when? Int J Technol Assess Health Care. 2012;28(1):29–35.
Shekelle PG, Ortiz E, Rhodes S, Morton SC, Eccles MP, Grimshaw JM, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461–7.
Vernooij RW, Sanabria AJ, Sola I, Alonso-Coello P, Martinez GL. Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks. Implement Sci. 2014;9:3.
Alonso-Coello P, Martinez Garcia L, Carrasco JM, Sola I, Qureshi S, Burgers JS. The updating of clinical practice guidelines: insights from an international survey. Implement Sci. 2011;6:107.
Martínez García L, Arevalo-Rodriguez I, Sola I, Haynes RB, Vandvik PO, Alonso-Coello P. Strategies for monitoring and updating clinical practice guidelines: a systematic review. Implement Sci. 2012;7:109.
Guidelines International Network. G-I-N Updating Guidelines Working Group. Scotland: Scottish Charity No: SC034047. Available from: http://www.g-i-n.net/working-groups/updating-guidelines.
Martínez García L, Pardo-Hernandez H, Superchi C, Niño de Guzman E, Ballesteros M, Ibargoyen Roteta N, et al. Methodological systematic review identified major limitations in prioritisation processes for updating. J Clin Epidemiol. 2017.
Martínez García L, Sanabria AJ, Araya I, Lawson J, Solà I, Vernooij RWM, et al. Efficiency of pragmatic search strategies to update clinical guidelines recommendations. BMC Med Res Methodol. 2015;15:57.
Gartlehner G, West SL, Lohr KN, Kahwati L, Johnson JG, Harris RP, et al. Assessing the need to update prevention guidelines: a comparison of two methods. Int J Qual Health Care. 2004;16(5):399–406.
Fochsen G, Zingmark P, Mild M. Living guidelines: A new model for guideline maintenance in Sweden. 13th G-I-N Conference; Philadelphia, USA; 2016.
Venhorst K, Ploegmakers M, Holtus A, Van Barnelveld T. What is needed to realize a dynamic process of guideline updating. 11th G-I-N Conference; Melbourne, Australia; 2014.
Robbins C, Tom G. Dynamic Updating of Clinical Practice Guidelines (CPGs). 9th G-I-N Conference; Berlin, Germany; 2012.
Zelman Lewis S. Sustainable Living Guidelines: A Model for the Future. 8th G-I-N Conference; Seoul, South Korea; 2011.
Johnston ME, Brouwers MC, Browman GP. Keeping cancer guidelines current: results of a comprehensive prospective literature monitoring strategy for twenty clinical practice guidelines. Int J Technol Assess Health Care. 2003;19(4):646–55.
Totten A, Zelman Lewis S, Patel S, Shorte V, Kolakowshy-Hayner S. Making the Transition to “Living Guidelines”: Challenges and Opportunities. 13th G-I-N Conference; Philadelphia, USA; 2016.
Grupo de trabajo de la Guía de práctica clínica de atención en el embarazo y puerperio. Guía de práctica clínica de atención en el embarazo y puerperio. Ministerio de Sanidad, Servicios Sociales e Igualdad. Agencia de Evaluación de Tecnologías Sanitarias de Andalucía; 2014. Guías de Práctica Clínica en el SNS: AETSA 2011/10.
Martínez García L, McFarlane E, Barnes S, Sanabria AJ, Alonso-Coello P, Alderson P. Updated recommendations: an assessment of NICE clinical guidelines. Implement Sci. 2014;9:72.
Vernooij RW, Alonso-Coello P, Brouwers M, Martinez GL. Reporting items for updated clinical guidelines: Checklist for the Reporting of Updated Guidelines (CheckUp). PLoS Med. 2017;14(1):e1002207.
Selva A, Sola I, Zhang Y, Pardo-Hernández H, Haynes RB, Martínez García L, Navarro T, Schünemann H, Alonso-Coello P, et al. Development and use of a content search filter for studies on how patients and other stakeholders value health outcomes. Health Qual Life Outcomes. 2017 [accepted].
Craig D, Rice S. NHS Economic Evaluation Database Handbook. 3rd ed. York: Centre for Reviews and Dissemination, University of York; 2007.
Vandvik PO, Brandt L, Alonso-Coello P, Treweek S, Akl EA, Kristiansen A, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.
Uhlig K, Berns JS, Carville S, Chan W, Cheung M, Guyatt GH, et al. Recommendations for kidney disease guideline updating: a report by the KDIGO Methods Committee. Kidney Int. 2016;89(4):753–60.
The authors would like to thank Maria Victoria Leo for her help in editing the final version of the manuscript.
The authors would like to acknowledge the members of the Pregnancy Clinical Guideline Updating Working Group: Laura Martínez García, Hector Pardo-Hernández, Andrea Juliana Sanabria, Pablo Alonso-Coello, Longinos Aceituno-Velasco, Ignacio Araya, Emilia Bailon Muñoz, Petra Díaz del Campo, Itziar Etxeandia-Ikobaltzeta, Manuel Fillol Crespo, Lucía Fuertes Luis, Elvira García Álvarez, Laura García Carrascosa, Natalia Gómez-Gómez, Blanca Herrera Crerizo, Jacinta Landa Goñi, Dario López, Arturo Louro González, María Dolores Martínez-Romero, Juliana Ester Martín-López, Adoración Montejo Ráez, Dimelza Osorio, Isabel Roman Romera, Flavia Salcedo-Fernandez, Anna Selva, Ivan Solà, Helen Strivens, Rafael Torrejón-Cardoso, Mar Trujillo-Martín, and Robin WM Vernooij.
This project is funded by research grants from the Instituto de Salud Carlos III through the project “PI14/02006” (co-funded by European Regional Development Fund/European Social Fund “Investing in your future”). LMG is funded by a Juan Rodés contract from the Instituto de Salud Carlos III (JR15/00004). PAC is supported by a Miguel Servet investigator contract from the Instituto de Salud Carlos III (CPII15/0034).
Availability of data and materials
The datasets generated and/or analysed during the current study are available from the corresponding author upon request.
LMG and PAC contributed to the conception and design of the study. The technical team participated in the data extraction and in the design and running of literature searches and in the data extraction. LMG, HPH, JAS, and PAC are responsible for the writing of the manuscript. All authors participated in the analysis and interpretation of data and contributed by commenting on and editing the manuscript. All authors read and approved the final version of the manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Martínez García, L., Pardo-Hernández, H., Sanabria, A.J. et al. Continuous surveillance of a pregnancy clinical guideline: an early experience. Syst Rev 6, 143 (2017). https://doi.org/10.1186/s13643-017-0506-7
- Diffusion of innovation
- Dissemination and implementation
- Evidence-based medicine
- Practice guidelines