- Open Access
- Open Peer Review
Guideline-based quality indicators—a systematic comparison of German and international clinical practice guidelines: protocol for a systematic review
Systematic Reviews volume 7, Article number: 5 (2018)
Quality indicators (QIs) are used in assessing the quality of healthcare. Evidence-based clinical practice guidelines (CPGs) are relevant sources for generating QIs. In this context, QIs are important tools to assess the implementation of guideline recommendations. However, the methodological approaches to guideline-based QI development vary considerably.
In Germany, the guideline classification scheme of the AWMF (German Association of the Scientific Medical Societies) differentiates between S1-, S2k-, S2e-, and S3-CPGs depending on the methodological approach. Thus, S3-CPGs are consensus- and evidence-based CPGs and have the highest methodological standard in Germany. An analysis of the status quo of reported QIs in S3-CPGs found 35 current S3-CPGs, which report 372 different QIs.
Currently, there is no gold standard for the development of guideline-based QIs. To our knowledge, no studies have investigated to what extent guideline-based QIs from different CPGs that are related to the same topic are consistent. The objective of this study is to compare guideline-based QIs and their underlying methodological approaches of German S3-CPGs with those of topic-related international CPGs.
Based on the previous identified German S3-CPGs (n = 35), which report quality indicators, we will conduct systematic searches in the guidelines databases of G-I-N (Guidelines International Network) and NGC (National Guideline Clearinghouse) to identify international CPGs matching the topics of the S3-CPGs. If necessary, we will search additionally the websites of the particular CPG providers for separate documents with regard to QIs. We will include evidence-based CPGs which report QIs. Reported QIs as well as methods of development and the rationale for QIs will be extracted and compared with those of the S3-CPGs.
This study will be part of the project “Systematic analysis of the translation of guideline recommendations into quality indicators and development of an evidence- and consensus-based standard,” supported by the German Research Association (DFG). The results of this analysis will feed into a subsequent qualitative study, which will consist of structured interviews with developers of international CPGs. Further, the results will be considered in a consensus study on standards of the translation of guideline recommendations into quality indicators in Germany.
Quality measurement and improvement play an important role in healthcare. For this purpose, quality indicators (QIs) can be used. There is no clear-cut definition of a QI. According to Lawrence and Frede, a QI is a “measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in the quality, of care provided” . The Joint Commission on Accreditation of Healthcare Organizations (JCAHO) defines QIs as “[…] quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes” . To be deemed as trustworthy and useful, QIs have to satisfy different criteria, such as relevance, validity, reliability, feasibility, and target group orientation [3,4,5,6]. To meet the high methodological requirements on QIs, they should be based where possible on scientific evidence and developed in a systematic and transparent way [7, 8].
As evidence-based clinical practice guidelines (CPGs) are designed to reflect current best practice, they are relevant sources for generating QIs [7, 9]. The term “guideline-based QIs” indicates in particular QIs that are either generated from already available CPGs or coupled with the process of CPG development . Besides assessing the quality of healthcare, these are important tools to assess the implementation of guideline recommendations [11,12,13]. However, the methodological approaches to guideline-based QI development vary considerably .
In Germany, the AWMF (German Association of the Scientific Medical Societies) provides the methodological framework for the development of CPGs by the scientific medical societies. The guideline classification scheme of the AWMF differentiates between S1-, S2k-, S2e-, and S3-CPGs depending on the methodological approach . Thus, S1-CPGs are based on an informal consensus building. In S2k-CPGs, a formal consensus method is applied in a representative panel, and S2e-CPGs include a systematic approach for literature searching as well as selection and appraisal of evidence. S3-CPGs comprise both the requirements for S2k-CPGs and those for S2e-CPGs and thus have the highest methodological standard in Germany. An analysis of the status quo of reported QIs in S3-CPGs from 2013 found 34 S3-CPGs, which report 394 different QIs (including measures of quality which are labeled such as “quality criteria” or “quality measure”) . For example, the S3-CPG “Diagnostics, treatment and follow-up care of malignant ovarial tumors” comprises 12 QIs, one of them regarding counseling by a social service (numerator: number of patients with counseling by a social service; denominator: all patients with an initial diagnosis of ovarian cancer and treatment in a clinical institution) . In the S3-CPG “Long-Term Opioid-Use in Non-Cancer Pain,” three QIs are stated, such as the QI “number of patients with somatoform pain disorder who are treated with an opioid” . An update (search up to 2016) of this analysis (not yet published) found 35 current S3-CPGs, which report 372 different QIs. Four S3-CPGs were developed by the National Program for Disease Management Guidelines (DMG), 15 by the German Guideline Program in Oncology (GGPO), and 16 by various medical societies. Particularly, the CPGs of the DMG and GGPO have a broad scope and cover various areas of medical care. For these CPGs, the development of guideline-based QIs is obligatory [11,12,13].
Although a working group of the Guidelines International Network (G-I-N) recently proposed a set of reporting standards for guideline-based performance measures , there is currently no gold standard for the development of guideline-based QIs [10, 19]. To our knowledge, no studies have investigated to what extent guideline-based QIs from different CPGs are consistent. Our hypothesis is that QIs from S3-CPGs are in many cases not corresponding with QIs of topic-related international CPGs.
The objective of this study is to compare guideline-based QIs and their underlying methodological approaches of the 35 previously identified German S3-CPGs with those of topic-related international CPGs.
Our study is not registered with PROSPERO as we will not report health-related outcomes.
CPGs will be included in this study when they meet the following criteria:
QIs are reported
The CPG is an evidence-based CPG
The topic and recommendations have to be comparable with those of at least one of the 35 previous identified S3-CPGs (see Additional file 1)
Country of CPG development belongs to WHO-Stratum A 
Date of publication: between 2012 and 2017
Published in German, English, French, Spanish, Dutch, Norwegian, or Swedish
Current full-text version is available at no charge
The validity date of the CPG, indicated by the CPG developer, is not exceeded
If QIs are solely reported in a separate document, which is not a supplement to the CPG (e.g., evidence or methodological report), they have to be explicitly linked with the particular CPG. Otherwise, we will assume that these QIs are not guideline-based, and we will exclude the guideline. An example for such a separate document that contains guideline-based QIs is a document from the website of the National Institute for Health and Care Excellence (NICE): “NICE menu of general practice and clinical commissioning group indicators” . The mentioned NICE-QIs usually are linked with particular CPGs (e.g., NICE guideline NG17). Evidence-based CPGs are defined in this analysis as guidelines whose recommendations are as follows:
Based on a systematic literature search
Clearly identifiable and with an assigned grade of recommendation (GoR) and/or a level of evidence (LoE)
Explicitly or implicitly linked to the references of the underlying evidence
Information sources and search strategy
Based on the previously identified S3-CPGs which report QIs, we will conduct systematic searches in the guidelines databases of G-I-N and NGC (National Guideline Clearinghouse) to identify international CPGs matching the topics of the S3-CPGs. The search strategies will include suitable keywords relating to the clinical topics and as appropriate truncations as well as Boolean operators. In cases where we cannot identify topical eligible guidelines, we will screen the websites of CPG providers additionally, whereby the searches will be tailored to the structure and capabilities of the websites. Furthermore, we will crosscheck the reference lists of the S3-CPGs and the international CPGs eligible for inclusion in the analysis.
In cases where topical eligible CPGs comprise neither QIs nor links to QIs, we will search the websites of the particular CPG providers for separate documents with regard to QIs that are explicitly linked with the particular CPG.
Data management and selection process
One reviewer will screen the titles of records, and the full texts of those deemed eligible for inclusion will be retrieved. In the next step, the screening of full texts will be conducted by one reviewer and checked by another. The reasons for exclusion will be documented, and any disagreements will be resolved through discussion and consensus.
The records will be uploaded and managed using Microsoft Excel.
In cases where no eligible CPG matching the topic of a S3-CPG can be found, we will exclude the particular CPG from analysis.
Data collection process and data items
A standardized extraction form will be developed based on the data extraction items used in a preliminary project  and pilot-tested. The following information will be collected:
Information on QI-development group (number of members and positions, such as methodologists, clinicians, patient representatives)
Labeling of the measure of quality, e.g., QI, quality criteria, performance measure
Categorization of QI in structure, process, outcome indicator according to the definition of Donabedian  (in case of missing assignment by the guideline authors an own assignment will be made)
Underlying recommendations, if the QIs are based explicitly or implicitly on those
Reported rationale for the QI
Reported measurement properties of QI, e.g., reliability and validity 
Reported intended purpose of QI, e.g., quality reporting, quality management systems, evaluation of CPGs
Reported quality objectives
Methods of QI-development, e.g., searches for existing QIs, consensus methods, assessment-tools
The extractions will be conducted by one reviewer and checked by another, any disagreements will be resolved through discussion and consensus.
As a high methodological quality of CPGs is asked to be a source of high quality and trustworthy guideline-based QIs [10, 18], the methodological quality of all included CPGs will be appraised using the domain “Methodological Rigor of Development” of the German Instrument for Methodological Guideline Appraisal (DELBI) . Seven items will be rated on a 4-point scale (whereby one = “strongly disagree,” two = “disagree,” three = “agree,” and four = “strongly agree”):
Systematic methods were used to search for evidence
The criteria for selecting the evidence are clearly described
The methods used for formulating the recommendations are clearly described
Health benefits, side effects, and risks have been considered in formulating the recommendations
There is an explicit link between the recommendations and the supporting evidence
The guideline has been externally reviewed by experts prior to its publication
A procedure for updating the guideline is provided
Two reviewers will perform the quality assessment independently. In case of two or more points of difference in the appraisal of the two reviewers, disagreement will be resolved through discussion and consensus. A domain score will be calculated by summing up the scores of the individual items and by standardizing the total as the percentage of the maximum possible score for the domain (4 (strongly agree) × 7 (items) × 2 (appraisers)) .
Reviewers who have been involved in the development of the included CPGs will not participate in their quality assessment.
Data synthesis will contain a descriptive analysis and a tabular comparison of the QIs of the included CPGs and those of the S3-CPGs for each clinical topic and when applicable for each underlying recommendation. We will collect the number of CPGs that give information to the QI-development group, the methods of QI-development, as well as the rationale and intended purpose of QI. On the basis of reported QIs, we will collect the number of QI for which quality objectives and measurement properties are reported as well as the number of QI that are explicitly or implicitly based on guideline recommendations.
For each matched pair of CPGs, we will compare the suggested QIs and assess if the QIs agree, disagree, or if they are not comparable. We will assign QIs on the same topic either to the category “not different/slightly different” or “different.” QIs that are not comparable will be extracted under the category “QI only defined in the international respectively the S3-CPG. For each category, we will collect the number of QIs respectively QI-pairs. Furthermore, the methods for QI-development will be summarized narratively.
This study will be part of the project “Systematic analysis of the translation of guideline recommendations into quality indicators and development of an evidence- and consensus-based standard,” supported by the German Research Association (DFG). It will be the second systematic analysis in the overall project. The results of this analysis will feed into a subsequent qualitative study which will consist of structured interviews with developers, methodologists, and users of international guidelines. Both studies intend to deliver additional information to existing research on methods for the development of guideline-based QIs [10, 18]. For the analysis of possible differences between QIs from different CPGs, we will consider existing guidelines or rather QI development manuals of the respective guideline organization.
An overview of the overall project is shown in Fig. 1.
Presenting and reporting the results
This protocol adheres to the “Preferred Reporting Items for Systematic Review and Meta-Analysis-Protocols (PRISMA-P)” . As PRISMA-P aims to guide the development of protocols for systematic reviews evaluating therapeutic efficacy, we deviated from the original checklist by omitting items (e.g., outcomes and prioritization) due to the methodological focus of our planned systematic review (see Additional file 2 for the completed PRISMA-P checklist).
The results of our study will be considered in the last phases of the overall project, namely a consensus-study on standards of the translation of guideline recommendations into quality indicators.
German Association of the Scientific Medical Societies
Clinical practice guideline
German Instrument for Methodological Guideline Appraisal
Guidelines International Network
National Guideline Clearinghouse
Preferred Reporting Items for Systematic Review and Meta-Analysis-Protocols
Lawrence M, Frede O. Indicators of quality in health care. Eur J Gen Pract. 1997;3:103–8.
The Joint Commission on Accreditation of Healthcare Organizations (JCAHO). Characteristics of clinical indicators. Qual Rev Bull. 1989;11:330–339.
Kelley Edward, Jermy H. Health care quality indicators project conceptual framework paper. 2006: OECD Health Working Papers: 23, https://www.oecd.org/els/health-systems/36262363.pdf. Accessed 08 May 2017.
Reiter A, et al. QUALIFY—a tool for assessing quality indicators. Z Arztl Fortbild Qualitatssich. 2007;101(10):683–8.
National Quality Forum, Measure evaluation criteria and guidance for evaluating measures for endorsement. 2016: http://www.qualityforum.org/Measuring_Performance/Endorsed_Performance_Measures_Maintenance.aspx. Assessed 08 May 2017.
Geraedts M, Selbmann HK, Ollenschlaeger G. Critical appraisal of clinical performance measures in Germany. Int J Qual Health Care. 2003;15(1):79–85.
Campbell SM, et al. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11(4):358–64.
Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care. 2003;15(6):523–30.
Institute of Medicine (U.S.). Committee on redesigning health insurance performance measures, P., and performance improvement programs, performance measurement: accelerating improvement (pathways to quality health care series). Washington: The National Academies Press; 2006. https://www.nap.edu/. Accessed 08 May 2017.
Kotter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators—a systematic review. Implement Sci. 2012;7:21.
German Guideline Program in Oncology (GGPO). Development of guideline based quality indicators. 2013: http://leitlinienprogramm-onkologie.de/uploads/tx_sbdownloader/QIDP_GGPO_2013.pdf. Accessed 08 May 2017.
German Medical Association (GMA), National Association of Statutory Health Insurance Physicians (NASHIP), and Association of the Scientific Medical Societies (AWMF), National Programme for Disease Management Guidelines. Method Report. 4th edition. 2010 http://www.versorgungsleitlinien.de/methodik/reports. Accessed 08 May 2017.
Nothacker MJ, Langer T, Weinbrenner S. Quality indicators for National Disease Management Guidelines using the example of the National Disease Management Guideline for “Chronic Heart Failure”. Z Evid Fortbild Qual Gesundhwes. 2011;105(1):27–37.
Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften (AWMF)- Ständige Kommission Leitlinien, AWMF-Regelwerk „Leitlinien". 2012: http://www.awmf.org/leitlinien/awmf-regelwerk.html. Accessed 08 May 2017.
Schmitt J, et al. Recommendations for quality indicators in German S3 guidelines: a critical appraisal. Gesundheitswesen. 2014;76(12):819–26.
Leitlinienprogramm Onkologie (Deutsche Krebsgesellschaft, Deutsche Krebshilfe, AWMF). S3-Leitlinie Diagnostik, Therapie und Nachsorge maligner Ovarialtumoren, Langversion 2.0. AWMF Registrierungsnummer: 032-035OL. 2016: http://leitlinienprogramm-onkologie.de/Leitlinien.7.0.html. Assessed 21 June 2017.
Deutsche Schmerzgesellschaft. S3 - Leitlinie „Langzeitanwendung von Opioiden bei nicht tumorbedingten Schmerzen - "LONTS". AWMF Registernummer: 145/003. 2015: http://www.awmf.org. Assessed 19 June 2017.
Nothacker M, et al. Reporting standards for guideline-based performance measures. Implement Sci. 2016;11:6.
Blozik E, et al. Simultaneous development of guidelines and quality indicators—how do guideline groups act? A worldwide survey. Int J Health Care Qual Assur. 2012;25(8):712–29.
World Health Organization, List of Member States by WHO Region and Mortality Stratum. http://www.who.int/choice/demography/mortality_strata/en/. Accessed 20 Oct 2017.
National Institute for Health and Care Excellence (NICE) (2016). The NICE menu of general practice and clinical commissioning group indicators. https://www.nice.org.uk/Media/Default/Standards-and-indicators/indicator-menu-update-aug-16.pdf. Assessed 08 Nov 2017.
Donabedian A. The role of outcomes in quality assessment and assurance. QRB Qual Rev Bull. 1992;18(11):356–60.
Mokkink LB, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.
German Association of the Scientific Medical societies (AWMF) and Agency for Quality in Medicine (ÄZQ), German Instrument for Methodological Guideline Appraisal 2008: http://www.delbi.de/. Accessed 08 May 2017.
Moher D, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.
We thank Max Geraedts (University of Marburg) for reviewing the final draft of the protocol.
German Research Association (DFG) (grant no. NE 385/23-1); the DFG was not involved in developing the protocol.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
MN was involved in the development of several S3-CPGs that are considered in this review. MB was involved in the preparation of the evidence report for one CPG that is considered in this review. MB, JB, MN, JS, EN, and DP were involved in the development of several S3-CPGs that are not considered in this review. DS and MS declare no competing interests (financial and non-financial).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Becker, M., Breuing, J., Nothacker, M. et al. Guideline-based quality indicators—a systematic comparison of German and international clinical practice guidelines: protocol for a systematic review. Syst Rev 7, 5 (2018) doi:10.1186/s13643-017-0669-2
- Quality indicator
- Performance measures
- Systematic review