- Open Access
Assessing the completeness and comparability of outcomes in systematic reviews addressing food security: protocol for a methodological study
Systematic Reviews volume 9, Article number: 9 (2020)
Systematic reviews should specify all outcomes at the protocol stage. Pre-specification helps prevent outcome choice from being influenced by knowledge of included study results. Completely specified outcomes comprise five elements: (1) domain (title), (2) specific measurement (technique/instrument), (3) specific metric (data format for analysis), (4) method of aggregation (how group data are summarised), and (5) time points. This study aims to assess the completeness of outcome pre-specification in systematic reviews of interventions to improve food security, specifically food availability, in low- and middle-income countries, as well as to assess the comparability of outcome elements across reviews reporting the same outcome domains.
We will examine systematic reviews from an ongoing overview of systematic reviews, which assessed the effects of interventions addressing food insecurity through improving food production, access, or utilisation compared with no intervention or a different intervention, on nutrition outcomes. We will examine the original protocols; if unavailable, we will examine the “Methods” section of the systematic reviews’ most recent version. One investigator will identify and group all outcome domains that the authors of the included protocols intended to measure in the systematic review and a second investigator will verify the domains. For outcome domains reported in at least 25% of protocols, one author will extract data using a pre-specified form and a second author will verify the data. We will use descriptive statistics to report the number, types, and degree of specification of outcomes in included protocols. We will assess the extent of completeness of outcome pre-specification based on the number of outcome elements (out of five). We will assess comparability of outcome domains through examining how individual elements are described across SRs reporting the same outcome domains.
Our findings will contribute to understanding about the best approach to pre-specify outcomes for systematic reviews and primary research in the field of food security.
Systematic reviews (SRs) summarise evidence to inform health decision-making. All outcomes important for patients, clinicians, and policy-makers should be pre-specified in SRs to avoid selective reporting of outcomes based on the results in the included studies. Furthermore, completeness of outcome specification may improve the utility of a SR for decision-makers.
In clinical trials, pre-specification of outcomes reduces the number of statistical analyses and thus the risk of type 1 error, i.e. the likelihood of finding a significant effect when there is no true effect. Pre-specification also decreases the risk of outcome reporting bias, i.e. selective or inadequate reporting of outcomes based on the strength or direction of the results. This bias has also been reported in SRs [1, 2]. Pre-specification of outcomes at the protocol stage, i.e. before data collection and analyses have been done, prevents decisions, such as which outcomes, which specific measurements, and which time points should be examined, from being influenced by the knowledge of the results . Although these are common issues in trials, they are also increasingly recognised as being important for SRs [4, 5].
SRs gather data from multiple primary studies, which often report data on the same outcome in different ways; pre-specification prevents SR authors from making post-hoc decisions regarding studies and outcomes for inclusion in the SR . Research has shown that trials assessing the same condition and intervention often report multiple different outcomes as they disagree on the important outcomes to measure, that outcomes selected across reviews on the same condition differ,  and that outcomes reported across trials and SRs on same condition have limited overlap [7, 8]. Mayo-Wilson and colleagues  found, in a sample of reports addressing the same intervention, that many trials were associated with more than one report and that some outcome domains had multiple definitions, with variations in measures, metrics, etc. Consequently, they found that there were often multiple results that could be associated with the same outcome domain, and thus different meta-analyses using different outcomes could change the magnitude and statistical significance of the meta-analysis and thus the conclusions of a SR. To reduce the risk of outcome reporting bias, it is recommended that outcomes are fully pre-specified in SRs [9, 10].
Cochrane recommends limiting the number of outcomes (ideally to no more than seven) in a SR and pre-specifying them . SR authors should focus on outcomes likely to be most meaningful to clinicians, patients, and policymakers and should interpret cautiously any indirect or surrogate outcomes (e.g. serum albumin for nutritional status). Sometimes, the patient-important and clinically meaningful outcomes for decision-making are not reported in primary studies included in a SR, and this would be an important gap in primary research to highlight at the SR level.
A common framework has been developed and used for defining outcomes and assessing the completeness of their pre-specification [6, 12, 13]. This framework is also now recommended as part of new Cochrane Handbook guidance . According to this framework, a completely specified outcome comprises five elements: (1) outcome domain or title, (2) specific measurement or technique/instrument used, (3) specific metric or format of outcome data that will be used for analysis, (4) method of aggregation or how data will be summarised, and (5) time points that will be used for analysis (Table 1). This useful and intuitive framework has been used to assess SRs, such as those produced by Cochrane Eyes and Vision  and Cochrane Wounds  Review Groups, but has not been used to assess SRs of interventions to address food security.
Food security exists when people have physical, social, and economic access to sufficient, safe, and nutritious food that meets their dietary needs and food preferences for an active and healthy life . A world without hunger and malnutrition is one of the United Nations Sustainable Development Goals (SDGs) . There is currently a rise in global hunger, particularly in Africa and South America; one in nine people in the world were undernourished in 2017 . Child undernutrition is a major public health burden, especially in low- and middle-income countries (LMICs), where 66% and 75% of all children under the age of 5 years who are stunted and wasted in 2017, respectively, live . Malnutrition resulting from hunger and food insecurity has detrimental consequences for individuals and societies. At the individual level, it leads to poor child development and health, increased risk of disability, morbidity, and mortality, and decreased productivity and income-generating potential in adults who suffered from malnutrition in early childhood [17,18,19]. At the societal level, malnutrition is associated with direct losses in economic productivity due to poor mental and physical performance and premature death, and with indirect losses due to decreased working and intellectual capacity of the working population, and to loss of resources due to increased health care costs . It is therefore important to address food insecurity, and SRs may be able to identify the most effective interventions.
Through the authors’ experience in conducting an overview of SR (ongoing) and a SR addressing food security-related questions , they identified multiplicity of outcome reporting in SRs and primary studies, which makes it difficult to harmonise and synthesise this evidence. Since pre-specification contributes to reducing reporting bias, we felt it was important to assess whether and to what extent authors of SRs addressing food security do this. Through this process, we will also be able to identify commonly reported outcomes across these SRs, which will contribute toward a planned next phase of identifying core outcome sets for food security. The development of core outcome sets is another recommendation for reducing outcome reporting bias in the SR process; it helps ensure that most appropriate review outcomes are selected and reported in primary studies, which will then inform systematic reviews . There has been some work looking at the importance of identifying core indicators for food security, as a more efficient way of measuring and reporting progress toward achieving food security that does not rely on long lists of indicators that are not easily comparable across time and space [22, 23]. However, existing guidance, although informative, is more geared toward country-level monitoring, or does not include all relevant outcome domains, and thus is not very useful to identify core outcomes to report at primary research and SR levels.
Therefore, we aim to use the outcomes definition framework described above to assess the completeness of outcome pre-specification in a set of SRs addressing food security. We also aim to assess the comparability of outcome elements across SRs reporting the same outcome domain. Harmonisation of outcomes across SRs addressing related questions is potentially useful for cases where overviews of SRs are conducted to inform decision-making .
This project aims to (1) describe all outcomes reported in SRs of interventions designed to improve food security in low- and middle-income countries (LMICs), (2) assess the completeness of outcome pre-specification in these reviews, and (3) assess the comparability of outcome elements across reviews reporting the same outcome domains. Our findings, once disseminated, can lead to better awareness of the importance of appropriate pre-specification and reporting of outcomes in SRs and primary research in this field. Additionally, our findings can inform a subsequent project aiming to identify core outcomes for this area.
Selection of protocols and SRs
For this study, we will include SRs that are being evaluated in an ongoing overview of SRs of interventions addressing food security. The eligibility criteria are:
Participants (P)—Any communities or groups of individuals in LMICs, across all life cycle stages;
Interventions (I)—Any intervention to address food insecurity through improvements in the production, access, or utilisation of food, or a combination of these;
Comparisons (C)—No intervention or a different intervention to address food insecurity;
Outcomes (O)—Any outcome related to food security, including nutritional status, measured at national or regional, district or community, or at household and individual levels.
Study Design (D)—SRs, defined as reviews that had pre-determined objectives, predetermined criteria for eligibility, searched at least two data sources of which one needed to be an electronic database, and performed data extraction and risk of bias assessment . Both protocols and completed systematic reviews were eligible.
Setting (S)—Included at least one study from a LMIC (according to the World Bank definition; https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups).
To identify SRs for the overview, we searched seven electronic databases from 1979 to February 2015: (1) MEDLINE (via PubMed), (2) the Cochrane Database of Systematic Reviews, (3) Database of Abstracts of Reviews of Effects (DARE) through the Cochrane Library, (4) Trip Database (which includes records in the EPPI library), (5) www.healthevidence.org, (6) Campbell library, and (7) 3ie database (more details regarding this overview are available on request).
We will seek the original protocols of the included SRs, through searching for links to the protocol in the included review, searching PROSPERO, and searching the Cochrane Library for the protocol version of the review (in the case of Cochrane Reviews). If necessary, we will also contact authors and Cochrane Review Group editors. If the original protocol is not available for a given SR, we will use the “Methods” section of the most recent version of the SR (if it had been updated). For simplicity, we hereafter refer to each SR as a “protocol.”
Selection of outcome domains
One investigator will identify and group all outcome domains that the authors of the included protocols intended to measure in the SR. To do so, we will review all sections of the protocols, not just the “Methods” sections. A second investigator will verify the definition and grouping of the domains identified. Discrepancies in outcome categorisation will be discussed by the author team. Of all outcome domains identified, we will select the key outcome domains as those that are reported in at least 25% of included protocols for data extraction.
Data items extracted from each protocol will include:
Publication year and first author (if the protocol was published) and date of protocol (if the protocol was not published).
Details regarding the intervention category (i.e. addressing food availability, access, or utilisation) and types of intervention within each category (e.g. agricultural interventions, income-generating interventions, cash transfers, nutrition education).
Data regarding the five elements of outcome pre-specification for each outcome [6, 12, 13]
Specific measurement used (this element will be considered specified if the SR authors stated how the outcome should be measured, including [where relevant] instruments, tools, scales, scores, and/or how the outcome should be defined),
Specific metric used (this element will be considered specified if the SR authors specified how they would analyse the data, including change from baseline, value at a time point, or time-to-event),
Method of aggregation used (this element will be considered specified if the SR authors specified how the data were to be summarised, including as mean, median, percentage or proportion, or an absolute number), and
Time points at which outcomes were measured or analysed (this element will be considered specified if the authors specified the time points to be used in their analysis).
Each outcome domain could have more than one outcome specification, in which case we will extract all specifications. Two authors will pilot the data extraction form using four protocols. Subsequently, one author will conduct data extraction for all protocols, and a second author will verify all extracted data. Any disagreements will be resolved through discussion.
We will report the number, types, and degree of specification of outcomes in included protocols using descriptive statistics. We will assess the extent of completeness of outcome pre-specification based on the number of outcome elements specified out of five possible elements. We will summarise the median and interquartile range (IQR) for the number of outcome elements specified for each included outcome domain and describe the elements in more detail for each domain. We will also report the frequency of categories of specific measurement, metric, method of aggregation, and time points for each outcome domain. To assess comparability of outcome domains, we will assess how individual elements are described across SRs reporting the same outcome domains.
This study is underway; SRs have been identified and full data extraction will begin.
If any changes are made to the protocol in the final research report, these will be marked clearly and justified. This research will contribute understanding about the pre-specification and reporting of outcomes in SRs addressing food security, which could potentially inform the identification of standard outcome measures that should be reported in this field.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on request.
Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2015;68(3):237–45.
Schmid CH. Outcome reporting bias: a pervasive problem in published meta-analyses. Am J Kidney Dis. 2017;69(2):172–4.
Deeks J, Higgins J, Altman D. Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins J, Green S, editors. Cochrane handbook for systematic reviews of interventions version 510 (updated March 2011): the Cochrane Collaboration; 2011.
Bender R, Bunce C Fau - Clarke M, Clarke M Fau - Gates S, Gates S Fau - Lange S, Lange S Fau - Pace NL, Pace Nl Fau - Thorlund K, et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008;61(9):857–65.
Page MJ, McKenzie Je Fau - Forbes A, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol. 2013;66(5):524–37.
Liu Z, Saldanha IJ, Margolis D, Dumville JC, Cullum NA. Outcomes in Cochrane systematic reviews related to wound care: an investigation into prespecification. Wound Repair Regen. 2017;25(2):292–308.
Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PW, K. D. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;S0895-4356(16):30541–8.
Saldanha IJ, Lindsley K, Do DV, Chuck RS, Meyerle C, Jones L, et al. Comparison of clinical trial and systematic review outcomes for the 4 most prevalent eye diseases. JAMA Ophthalmol. 2017;135(9):933–40.
Mayo-Wilson E, Li T, Fusco N, Bertizzolo L, Canner JK, Cowley T, et al. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol. 2017;91:95–110.
Li T, Higgins JPT, (editors) DJ. Chapter 5: Collecting data. In: In: Higgins JPT TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). editor. Cochrane Handbook for Systematic Reviews of Interventions version 60 (updated July 2019). Cochrane; 2019. Available from www.training.cochrane.org/handbook. Accessed 4 November 2019.
O’Connor D, Green S, Higgins JPT. Chapter 5: Defining the review question and developing criteria for including studies. Version 5.1.0 (updated March 2011). In: Higgins JPTGS, editor. Cochrane Handbook of Systematic Reviews of Intervention: The Cochrane Collaboration; 2011.
Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400.
Zarin DA, Tse T Fau - Williams RJ, Williams Rj Fau - Califf RM, Califf Rm Fau - Ide NC, Ide NC. The ClinicalTrials.gov results database--update and key issues. N Engl J Med. 2011;364(9):852–60.
FAO, IFAD, UNICEF, WFP, WHO. The state of food security and nutrition in the world 2018. Building climate resilience for food security and nutrition. Rome; 2018.
United Nations. The sustainable development goals report. New York; 2018.
United Nations Children’s Fund (UNICEF), World Health Organization, International Bank for Reconstruction and Development/The World Bank. Levels and trends in child malnutrition: key findings of the 2018 Edition of the Joint Child Malnutrition Estimates. Geneva: World Health Organization; 2018.
Ecker O, Breisinger C. The food security system. A new conceptual framework. International Food Policy Research Institute; 2012.
Black RE, Victora CG, Walker SP, Bhutta ZA, Christian P, de Onis M, et al. Maternal and child undernutrition and overweight in low-income and middle-income countries. Lancet. 2013;382(9890):427–51.
Victora CG, Adair L Fau - Fall C, Fall C Fau - Hallal PC, Hallal Pc Fau - Martorell R, Martorell R Fau - Richter L, Richter L Fau - Sachdev HS, et al. Maternal and child undernutrition: consequences for adult health and human capital. Lancet. 2008;371(9609):340–57.
Durao S, Schoonees A, Ramokolo V, Oliveira JMD, Kristjansson E. Community-level interventions for improving access to food in low- and middle-income countries. Cochrane Database Syst Rev. 2015, Issue 2. Art. No.:CD011504. https://doi.org/10.1002/14651858.CD011504.
Kirkham JJ, Altman DG, Chan AW, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ. 2018;362:k3802.
FAO. Selecting a core set of indicators for monitoring global food security: a methodological proposal. Working Paper Series. Rome: Food and Agriculture Organization of the United Nations; 2014.
Carletto C, Zezza A, Banerjee R. Towards better measurement of household food security: harmonizing indicators and the role of household surveys. Global Food Security. 2013;2(1):30–40.
Young T, Rohwer A, Volmink J, Clarke M. What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS One. 2014;9(1):e86706
No specific funding was obtained for this study.
Ethics approval and consent to participate
As we are using secondary, already published, and de-identified data, ethics approval is not required.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Durão, S., Visser, M., Kredo, T. et al. Assessing the completeness and comparability of outcomes in systematic reviews addressing food security: protocol for a methodological study. Syst Rev 9, 9 (2020). https://doi.org/10.1186/s13643-019-1268-1
- Outcome pre-specification
- Outcomes reporting bias
- Systematic review methods
- Food security