- Open Access
Critical appraisal of clinical practice guidelines for the management of COVID-19: protocol for a systematic review
Systematic Reviews volume 10, Article number: 317 (2021)
In December 2019, a novel coronavirus, severe acute respiratory syndrome coronavirus 2 was identified as the cause of an acute respiratory disease, coronavirus disease 2019 (COVID-19). Given the lack of validated treatments, there is an urgent need for a high-quality management of COVID-19. Clinical practice guidelines (CPGs) are one tool that healthcare providers may use to enhance patient care. As such, it is necessary that they have access to high-quality evidence-based CPGs upon which they may base decisions regarding the management and use of therapeutic interventions (TI) for COVID-19. The purpose of the proposed study is to assess the quality of CPGs that make management or TI recommendations for COVID-19 using the AGREE II instrument.
The proposed systematic review will identify CPGs for TI use and/or the management of COVID-19. The MEDLINE, EMBASE, CINAHL, and Web of Science databases, as well as the Guidelines International Network, National Institute for Health and Clinical Excellence, Scottish Intercollegiate Guidelines Network, and the World Health Organization websites, will be searched from December 2019 onwards. The primary outcome of this study is the assessed quality of the CPGs. The quality of eligible CPGs will be assessed using the Appraisal of Guidelines, Research and Evaluation II (AGREE II) instrument. Descriptive statistics will be used to quantify the quality of the CPGs. The secondary outcomes of this study are the types of management and/or TI recommendations made. Inconsistent and duplicate TI and/or management recommendations made between CPGs will be compared across guidelines. To summarize and explain the findings related to the included CPGs, a narrative synthesis will also be provided.
The results of this study will be of utmost importance to enhancing clinical decision-making among healthcare providers caring for patients with COVID-19. Moreover, the results of this study will be relevant to guideline developers in the creation of CPGs or improvement of existing ones, researchers who want to identify gaps in knowledge, and policy-makers looking to encourage and endorse the adoption of CPGs into clinical practice. The results of this review will be published in a peer-reviewed journal and presented at conferences.
Systematic review registration
International Prospective Register for Systematic Reviews (PROSPERO)—CRD42020219944
In December 2019, a novel coronavirus, severe acute respiratory syndrome coronavirus 2 was identified as the cause of an acute respiratory disease, coronavirus disease 2019 (COVID-19) . COVID-19 was subsequently declared a pandemic by the World Health Organization on March 11, 2020 . Patients with COVID-19 are typically adults who present with symptoms such as fever, dry cough, myalgia, dyspnea, and fatigue . In contrast, severe cases may present with viral pneumonia, severe acute respiratory distress syndrome, and death . In November 2021, there were over 260 million cases of COVID-19 and more than 5.18 million deaths globally . Given the lack of successful treatments , there is an urgent need for a high-quality management of COVID-19.
As a result of the relative novelty of COVID-19, there is a lack of data to help guide prognosis. As such, healthcare providers require high-quality evidence-based tools that can support clinical decision-making and inform medical practice. Clinical practice guidelines (CPGs) are one such tool that healthcare professionals may use to guide clinical practice. CPGs are developed systematically and formulate recommendations based on the highest quality of supporting evidence . They provide healthcare professionals with explicit recommendations on how to proceed and help improve the consistency of care . In response to this pandemic, CPGs have been developed to guide the management of adult patients with COVID-19. High-quality, evidence-based CPGs have the potential to enhance the care of adults with COVID-19; however, this depends on the quality of the CPG and its adoption in clinical practice. Several studies have found that CPGs vary in quality for various health conditions [7,8,9], and consequently, may affect the healthcare providers’ ability to deliver high-quality care and service. Therefore, in order to enhance care, it is crucial that healthcare providers have access to high-quality evidence-based CPGs upon which they may base decisions regarding the management and use of therapeutic interventions (TIs) for COVID-19.
With an abundance of CPGs available for use by healthcare professionals, a strategy is needed to assess the quality of CPGs, so that those of highest quality are applied in the treatment and management of COVID-19 patients. One well-established and widely used instrument for assessing the rigor and methodological quality of CPGs is known as the Appraisal of Guidelines, Research and Evaluation II (AGREE II) instrument . As a tool for guideline appraisal, the AGREE II instrument seeks to inform the type of information found in CPGs, methodological considerations for guideline developers, and how likely appraisers are to recommend the use of the CPG in professional practice . In particular, the AGREE II instrument is used to assess CPGs across the following six domains: (1) scope and purpose, (2) stakeholder involvement, (3) rigor of development, (4) clarity of presentation, (5) applicability, and (6) editorial independence. By assessing the quality of CPGs across these unique domains, the AGREE II instrument can predict the outcomes associated with the implementation of a CPG, which can greatly inform healthcare practices as they pertain to COVID-19 treatment and management .
To date, few studies have examined the quality of CPGs for COVID-19 using the AGREE II instrument [11,12,13,14,15]. These studies found that the recommendations formulated within the CPGs were lacking evidence and that the overall quality of CPGs was poor and variable [11,12,13,14,15]. However, these studies are limited for several specific reasons. Dagens et al. only assessed CPGs for COVID-19 which were published by March 2020, soon after the first confirmed case of COVID-19 ; however, the quality of CPGs may have changed since March 2020. Since March 2020, there has been significant growth in COVID-19 research , as well as additional time to implement quality improvements in new CPGs for COVID-19. A study by Ong et al. limited its scope to only assessing CPGs on the perioperative anesthetic management of COVID-19 . Similarly, a study by Li et al. limited its scope to only evaluating CPGs related to the treatment of COVID-19 using Chinese herbal medicine . Another study by Yeo et al. limited its scope by population, as it only assessed the quality of CPGs pertinent to the management of COVID-19 in neonates born to mothers with COVID-19 . Due to the limitation in scope, these studies may have obtained results that do not accurately reflect the quality of all CPGs related to the management and/or treatment of COVID-19. Finally, one study by Wang et al. investigated the quality of evidence-based and consensus-based CPGs for the management of COVID-19; however, this study did not differentiate between the quality of CPGs for the management of adults and the quality of CPGs for pediatric populations .
To our knowledge, this study will be the first to assess the quality of evidence-based CPGs focused on the treatment and/or management of COVID-19 in adult patients. In doing so, the study will inform healthcare providers on which CPGs should be trusted the most and inform CPG developers of areas that need improvement. Hence, the primary objective of this study is to comprehensively assess the quality of evidence-based CPGs that make management or TI recommendations for adult patients with COVID-19 using the AGREE II instrument. The secondary objective is to determine the types of management and/or TI recommendations made, as well as which recommendations are inconsistent, unique, or duplicate across CPGs.
The present study will use the AGREE II instrument for the quality assessment of CPGs focused on TI use and/or the management of adult patients with COVID-19 . The present protocol was prospectively registered on the international Prospective Register of Systematic Reviews (PROSPERO) with the following registration number: CRD42020219944. The present protocol has been reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)  (see PRISMA-P checklist in Additional file 1). The completed systematic review will be reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) . If significant deviations are made from this protocol, then they will be reported and published with the results of the review.
The criteria for eligible COVID-19 CPGs were developed in accordance with the Population, Intervention, Comparison and Outcomes (PICO) framework . A CPG is a systematically developed guidance document that makes recommendations for a specific condition based on an analysis of benefits and risks of each intervention . Alternatively, when the evidence for interventions are limited such that an assessment of benefits and harms cannot be accurately made, consensus statements may be developed during a convention of experts . Recommendations are made in a consensus statement if there is sufficient agreement between the expert panel regarding the interventions; however, these recommendations may be biased by those involved in the recommendation development process . Despite these differences between CPGs and consensus statements, both are types of guidance documents . We will include CPGs that focus on adults (18 years or older) with COVID-19. With respect to the interventions, we will include CPGs that primarily discuss and make recommendations for TI use and/or the management of COVID-19. In terms of the TIs that are eligible, we will include all TIs used for the treatment and/or management of COVID-19. These include, but are not limited to, tocilizumab, recombinant angiotensin-converting enzyme 2, remdesivir, lopinavir/ritonavir, anticoagulants, antiviral combination therapies, hydroxychloroquine/chloroquine compounds, colchicine, corticosteroids, oseltamivir/amantadine, and convalescent plasma. We will not limit the inclusion of CPGs to specific TIs, as doing so would prevent a comprehensive assessment of CPGs focused on the management and/or TI use for COVID-19. We will include CPGs that are evidence-based, publicly available, published after 2019, and in English. We will exclude articles that focus on the diagnosis or screening of COVID-19 and the associated SARS-2-CoV-2 virus. COVID-19 diagnosis is commonly done through viral tests such as antigen testing immunoassays and nucleic acid amplification tests . Antigen immunoassays are often employed as a screening tool as they are easy to produce, detect the presence of specific antigen, and indicate current infection . In contrast, nucleic acid amplification tests are viral diagnostic tests that detect the presence of viral genetic material by identifying the ribonucleic acid comprising its genome [23, 24]. CPGs that are not in English or have a more recent update available from each organization will also be excluded. Furthermore, we will also exclude articles that are abstracts, editorials, letters, position papers, protocols, consensus statements, conference proceedings, and summaries of CPGs.
Outcomes and prioritization
The primary outcome of this study is the assessed quality of eligible CPGs. The assessed quality will be determined through the calculation of the scaled domain percentages for each CPG, as outlined in the AGREE II instrument . We will also report the average appraiser scores and average overall assessments, in addition to the scaled domain percentages. The secondary outcomes of this study are the types of management and/or TI recommendations made. We will determine which recommendations are inconsistent, unique, or duplicate across CPGs.
Search methods for identification of studies
Electronic database search
We will search databases including Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica Database (EMBASE), Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Web of Science from December 2019 onwards. As the databases have differing capabilities, we will adapt the search for each accordingly. The searches will use appropriate Medical Subject Headings (MeSH) terms to ensure inclusion of relevant titles and abstracts . We will also search the websites of guideline developing organizations for the most recent guidelines that satisfy the eligibility criteria. A sample search strategy to be run on MEDLINE is presented in Additional file 2. This search strategy may be modified to increase specificity and sensitivity.
Other data sources
We will search for relevant documents within CPG organizations, including Guidelines International Network (https://www.g-i-n.net/), National Institute for Health and Clinical Excellence (https://www.nice.org.uk/), Scottish Intercollegiate Guidelines Network (https://www.sign.ac.uk/), and the World Health Organization (https://www.who.int/), as well as grey literature sources.
Study selection process
Two reviewers will independently perform title and abstract screening using Rayyan . Articles that meet the eligibility criteria will be included and their respective full-texts will be retrieved for full-text screening. Full-text screening will be similarly performed independently and in duplicate. Any discrepancies regarding the eligibility of an article will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. Supplementary materials and related documents for CPGs that pass full-text screening will be retrieved thereafter. A PRISMA flow chart illustrating the details of the selection process will be provided .
Data collection and extraction
The reviewers will independently and in duplicate perform data collection and extraction in data extraction spreadsheets prepared a priori. Any discrepancies in the data extracted will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. For eligible CPGs, we will data extract demographic information including the author, title, year of publication, publishing organization, country, and funding source. Any management and/or TI recommendations made will also be extracted in addition to their corresponding level of evidence and grade of recommendation.
We will assess eligible CPGs using the AGREE II instrument which is a well-established and widely-validated international tool used to assess the quality and reporting of CPGs . It is composed of 23 items over six quality domains which include the following: (1) scope and purpose, (2) stakeholder involvement, (3) rigor of development, (4) clarity and presentation, (5) applicability, and (6) editorial independence . The instrument uses a seven-point Likert scale that ranges from strongly disagree (1) to strongly agree (7) that each item is met . Each eligible CPG will be evaluated by four appraisers as recommended by the AGREE II instrument . Prior to assessing the CPGs, each reviewer will complete the AGREE II Tutorial + Practice Exercise on the ‘My AGREE PLUS’ platform . The reviewers will also perform a minimum of three rounds of calibration with CPGs that vary in quality. The reviewers will then score the CPGs for each of the 23 items over 6 domains. The overall quality of the CPG (1 to 7) will sequentially be evaluated by the reviewers and used to recommend for or against the use of a CPG. Any major discrepancies in the scoring of a CPG (greater than 3 points) will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. We will then calculate the average overall assessment scores and average appraisal scores for each CPG. The average overall assessment scores will be calculated by averaging the scores each appraiser gave for ‘overall guideline assessment.’ The average appraisal scores will be calculated by averaging the score an appraiser gave for all 23 items, and then subsequently averaging this value calculated for each of the 4 appraisers. The quality of each domain will be compared within and across CPGs using the scaled domain percentages. The scaled domain percentages will be calculated through the summation of the appraisers’ scores for the items within each domain, and subsequently scaling this value as a percentage of the maximum possible score for the given domain.
Data synthesis and analysis
Descriptive statistics will be used to quantify the quality of the CPGs. Specifically, we will calculate the average overall assessment scores, average appraisal scores, and the standard deviation for each of these two scores. The scaled domain percentages will also be calculated. We will tabulate these results for ease of quality comparison both across and within the domains of the CPGs. A narrative synthesis will be provided to explain the characteristics and findings of the included CPGs.
Description and comparison of recommended interventions
We will tabulate the recommendations made in each CPG for the management of COVID-19. Additionally, we will compare inconsistent and duplicate TI and/or management recommendations made between CPGs. Specifically, we will indicate whether a therapy was recommended for or against use in a table. If a therapy was recommended, then it will be indicated with a green box. If a therapy is recommended against, then it will be indicated with a red box. If a therapy recommendation is unclear or conflicting, then it will be indicated with a yellow box. This will allow for ease of comparison between therapy recommendations across the eligible CPGs. In addition to comparing the CPGs in a table, we will also provide a narrative synthesis to describe and compare the recommendations of the included CPGs.
COVID-19 has affected over 260 million people and accounted for more than 5.18 million deaths globally . Classified as a pandemic by the World Health Organization, individuals with COVID-19 suffer from symptoms such as fever, dry cough, myalgia, dyspnea, and fatigue [1, 2], where critical cases may result in severe acute respiratory distress syndrome and death . Many of those individuals affected must turn to healthcare professionals for the high-quality care and management of the condition. Evidently, with the current lack of validated treatments, there is a greater emphasis placed on the management of COVID-19. As such, it is necessary that healthcare professionals have access to high-quality evidence-based CPGs which contain recommendations pertaining to the use of TIs and the management of COVID-19. The purpose of this systematic review will be to investigate the quality of existing CPGs designed for the management of adult patients with COVID-19 using the AGREE II instrument.
Strengths and limitations
There are several notable strengths to this study, including the use of a comprehensive systematic review to determine eligible CPGs pertaining to TI use and the management of COVID-19, use of multiple appraisers, and the use of the well-established and widely-validated AGREE II instrument.
There also exist some limitations inherent in our proposed study. Firstly, CPGs that are not in English should be excluded; however, this may result in the omission of CPGs pertinent to the proposed research question. Secondly, the target population of the CPGs for this study is all adults over 18 years of age; however, we do not intend on separately analyzing quality between CPGs for adults (18–65 years) from CPGs for elderly adults (65 years and above). It is possible that CPG quality may differ between these target populations. Lastly, given the COVID-19 pandemic, CPG developers may seek to rapidly disseminate knowledge to better manage COVID-19. As a result, CPG developers may prioritize the rapid dissemination of new CPGs instead of prioritizing the methodological quality of CPGs.
Our proposed systematic review and assessment will aide healthcare professionals who wish to improve the standard of care for patients with COVID-19 by incorporating CPGs into their practice, guideline developers in the creation of CPGs or improvement of existing ones, researchers who want to identify gaps in knowledge, and policy-makers looking to encourage and endorse the adoption of CPGs into clinical practice. Additionally, the evidence gathered through this systematic review will provide insight into gaps in available CPGs which may result in the identification of other research avenues.
Any amendments made to the proposed study will be indicated on PROSPERO and in the “Methods” section of the completed systematic review. This systematic review will be disseminated in a peer-reviewed journal and may be presented at local, national, or international conferences.
Availability of data and materials
- AGREE II:
Appraisal of Guidelines for Research & Evaluation II
Coronavirus Disease 2019
Clinical practice guideline
Patients, Intervention, Comparison and Outcomes
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols
Rothan HA, Byrareddy SN. The epidemiology and pathogenesis of coronavirus disease (COVID-19) outbreak. J Autoimmun. 2020;109:102433. https://doi.org/10.1016/j.jaut.2020.102433.
Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020;395:497–506. https://doi.org/10.1016/S0140-6736(20)30183-5.
Lovato A, De Filippis C. Clinical presentation of COVID-19: a systematic review focusing on upper airway symptoms. Ear Nose Throat J. 2020;99:569–76. https://doi.org/10.1177/0145561320920762.
COVID-19 Map [Internet]. Johns Hopkins Coronavirus Resource Center. https://coronavirus.jhu.edu/map.html. Accessed 26 Nov 2021.
Sanders JM, Monogue ML, Jodlowski TZ, Cutrell JB. Pharmacologic treatments for coronavirus disease 2019 (COVID-19). JAMA. 2020. https://doi.org/10.1001/jama.2020.6019.
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (Clinical research ed). 1999;318(7182):527–30. https://doi.org/10.1136/bmj.318.7182.527.
Burda BU, Norris SL, Holmer HK, Ogden LA, Smith ME. Quality varies acrossclinical practice guidelines for mammography screening in women aged40-49 years as assessed by AGREE and AMSTAR instruments. J Clin Epidemiol. 2011;64(9):968–76.22.
de Haas ER, de Vijlder HC, van Reesema WS, van Everdingen JJ, NeumannHA. Quality of clinical practice guidelines in dermatological oncology. J Eur Acad Dermatol Venereol. 2007;21(9):1193–8.23.
Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, Philip T, et al. International assessment of the quality of clinical practice guidelines inoncology using the appraisal of guidelines and research and evaluation instrument. J Clin Oncol. 2004;22(10):2000–7.
Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. Can Med Assoc J. 2010;182(18):E839–42. https://doi.org/10.1503/2Fcmaj.090449.
Dagens A, Sigfrid L, Cai E, Lipworth S, Cheng V, Harris E, et al. Scope, quality, and inclusivity of clinical guidelines produced early in the covid-19 pandemic: rapid review. BMJ. 2020:M1936. https://doi.org/10.1136/bmj.m1936.
Ong S, Lim WY, Ong J, Kam P. Anesthesia guidelines for COVID-19 patients - a narrative review and appraisal. Kor J Anesthesiol. 2020. https://doi.org/10.4097/kja.20354.
Li YX, Li J, Zhang Y, Tian YP, Zhang YG, Jin RJ, et al. Clinical practice guidelines and experts’ consensuses for treatment of coronavirus disease 2019 (COVID-19) patients with Chinese herbal medicine: a systematic review. Chin J Integr Med. 2020;26(10):786–93. https://doi.org/10.1007/s11655-020-3431-x.
Yeo KT, Oei JL, De Luca D, Schmölzer GM, Guaran R, Palasanthiran P, et al. Review of guidelines and recommendations from 17 countries highlights the challenges that clinicians face caring for neonates born to mothers with COVID-19. Acta Paediatr. 2020;109:2192–207. https://doi.org/10.1111/apa.15495.
Wang YY, Huang Q, Shen Q, Zi H, Li BH, Li MZ, et al. Quality of and recommendations for relevant clinical practice guidelines for COVID-19 management: a systematic review and critical appraisal. Front Med. 2021;8:703. https://doi.org/10.3389/fmed.2021.630765.
Haghani M, Bliemer MC, Goerlandt F, Li J. The scientific literature on coronaviruses, COVID-19 and its associated safety-related research dimensions: a scientometric analysis and scoping review. Saf Sci. 2020;129:104806. https://doi.org/10.1016/j.ssci.2020.104806.
Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349(jan02 1):g7647. https://doi.org/10.1136/bmj.g7647.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Bmj. 2021:372. https://doi.org/10.1136/bmj.n71.
Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Making. 2007;7(1):16. https://doi.org/10.1186/1472-6947-7-16.
Joshi GP, Benzon HT, Gan TJ, Vetter TR. Consistent definitions of clinical practice guidelines, consensus statements, position statements, and practice alerts. Anesth Analg. 2019;129(6):1767–70. https://doi.org/10.1213/ANE.0000000000004236.
Malherbe P, Smit P, Sharma K, McCaul M. Guidance we can trust? The status and quality of prehospital clinical guidance in sub-Saharan Africa: a scoping review. African J Emerg Med. 2021;11(1):79–86. https://doi.org/10.1016/j.afjem.2020.08.005.
Overview of testing for SARS-COV-2 (COVID-19) [Internet]. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/hcp/testing-overview.html. Accessed 26 Nov 2021.
Interim guidance for antigen testing for SARS-COV-2 [Internet]. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antigen-tests-guidelines.html. Accessed 26 Nov 2021.
Nucleic acid amplification tests (NAATs) [Internet]. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/lab/naats.html. Accessed 26 Nov 2021.
Lowe HJ, Barnett GO. Understanding and using the medical subject headings (MeSH) vocabulary to perform literature searches. Jama. 1994;271(14):1103–8. https://doi.org/10.1001/jama.1994.03510380059038.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.
My AGREE PLUS [Internet]. AGREE Enterprise website. https://www.agreetrust.org/resource-centre/agree-plus/. Accessed 26 Nov 2021.
The authors of this research declare that they have received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Ethics approval and consent to participate
The study will not require ethics approval. The results of this study will be published in a peer-reviewed journal and may be presented at local, national, or international conferences.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols 2015 Statement (PRISMA-P checklist): contains items that are required to be reported in a systematic review protocol.
Sample MEDLINE search for COVID-19 Clinical Practice Guidelines executed using the OVID interface: contains sample search strategy on MEDLINE database.
About this article
Cite this article
Dhivagaran, T., Abbas, U., Butt, F. et al. Critical appraisal of clinical practice guidelines for the management of COVID-19: protocol for a systematic review. Syst Rev 10, 317 (2021). https://doi.org/10.1186/s13643-021-01871-7
- Systematic review protocol
- AGREE II
- Critical appraisal
- Clinical practice guideline
- Guideline assessment