- Open Access
Evidence-based mapping of design heterogeneity prior to meta-analysis: a systematic review and evidence synthesis
Systematic Reviews volume 3, Article number: 80 (2014)
Assessment of design heterogeneity conducted prior to meta-analysis is infrequently reported; it is often presented post hoc to explain statistical heterogeneity. However, design heterogeneity determines the mix of included studies and how they are analyzed in a meta-analysis, which in turn can importantly influence the results. The goal of this work is to introduce ways to improve the assessment and reporting of design heterogeneity prior to statistical summarization of epidemiologic studies.
In this paper, we use an assessment of sugar-sweetened beverages (SSB) and type 2 diabetes (T2D) as an example to show how a technique called ‘evidence mapping’ can be used to organize studies and evaluate design heterogeneity prior to meta-analysis.. Employing a systematic and reproducible approach, we evaluated the following elements across 11 selected cohort studies: variation in definitions of SSB, T2D, and co-variables, design features and population characteristics associated with specific definitions of SSB, and diversity in modeling strategies.
Evidence mapping strategies effectively organized complex data and clearly depicted design heterogeneity. For example, across 11 studies of SSB and T2D, 7 measured diet only once (with 7 to 16 years of disease follow-up), 5 included primarily low SSB consumers, and 3 defined the study variable (SSB) as consumption of either sugar or artificially-sweetened beverages. This exercise also identified diversity in analysis strategies, such as adjustment for 11 to 17 co-variables and a large degree of fluctuation in SSB-T2D risk estimates depending on variables selected for multivariable models (2 to 95% change in the risk estimate from the age-adjusted model).
Meta-analysis seeks to understand heterogeneity in addition to computing a summary risk estimate. This strategy effectively documents design heterogeneity, thus improving the practice of meta-analysis by aiding in: 1) protocol and analysis planning, 2) transparent reporting of differences in study designs, and 3) interpretation of pooled estimates. We recommend expanding the practice of meta-analysis reporting to include a table that summarizes design heterogeneity. This would provide readers with more evidence to interpret the summary risk estimates.
Meta-analyses, which are quantitative methods for pooling results from epidemiologic studies, inform research priorities and health policy. Combining similar studies asking a similar research question is fundamental to the interpretability of summary risk estimates . Combining results in a meta-analysis from studies that are designed to answer different scientific questions may lead to imprecise and possibly invalid inferences [2, 3].
An assessment of the similarity of studies (that is, design heterogeneity) is a fundamental element of a meta-analysis of epidemiological studies [3–8]. There are two major types of heterogeneity: statistical heterogeneity and design heterogeneity (sometimes referred to as clinical and methodological diversity) . Statistical heterogeneity is purely a mathematical assessment; evidence of statistical heterogeneity indicates that there is greater statistical variance between the study results than would be expected by chance if the effect size was similar across studies [8, 10]. Design heterogeneity, in contrast, involves the extent to which the studies being considered for inclusion in a meta-analysis differ in study design, including population studied, specificity of exposure measurement, uniformity of diagnostic criteria (in the outcome), confounders measured, concomitant exposures measured, and statistical models [3, 7].
Reviews of the practice of meta-analysis in observational epidemiology have observed that investigators often emphasize the summarization function over the assessment of heterogeneity [2, 11]. Additionally, in a systematic overview of meta-analyses, we found fewer than a third of 47 eligible meta-analyses of lifestyle and dietary risk factors for type 2 diabetes (T2D) reported a detailed characterization of design heterogeneity that was used to guide the quantitative pooling of study results (manuscript in preparation). In contrast, more than 90% of the meta-analyses reported some assessment of statistical heterogeneity (Q statistic or I2 index). These observations illustrate that the assessment of design heterogeneity frequently occurs after statistical heterogeneity has been identified. In practice, design heterogeneity assessment would be informative if undertaken before any quantitative summarization takes place .
In 2013, two journals focusing on research synthesis methods (Systematic Reviews and Research Synthesis Methods) emphasized the importance of qualitative evaluation of studies selected for meta-analysis, calling for more strategies to aid conduct and reporting [12, 13]. In this paper, we present a strategy for objectively and transparently characterizing design heterogeneity of epidemiologic studies prior to meta-analysis.
Evidence-based mapping was used as a tool to diagram and tabulate data across a group of studies selected for meta-analysis, with the following three primary objectives:
to document differences in exposure (intervention), comparator, outcome, and study design and population characteristics;
to assess the design features and population characteristics associated with specific definitions of the exposure (intervention), comparator and outcome; and
to evaluate the diversity in modeling strategies (for example, assessment of confounding for observational epidemiology studies) and suggest simple summary measures to benchmark susceptibility of the exposure risk estimate to the influences of included (and excluded) co-variables in multivariable regression models.
We sought to summarize the detailed work of multiple evidence maps created to meet these objectives into a single table with a universal adaptable format. The aim of this table is to facilitate the reporting of design heterogeneity, which is fundamental to developing a protocol, analyzing data, and interpreting meta-analyses.
Evidence maps are a relatively new tool used to transparently generate a clear visual depiction of complicated data, either in the form of a diagram or a table . Evidence maps have been used to set research priorities by displaying existing research landscapes without linking study designs to study results [14–23]. Precisely because evidence mapping seeks to organize studies without summarizing results, they are natural tools for assessing design heterogeneity prior to meta-analysis. Therefore, we expanded evidence mapping methods by demonstrating their usefulness in planning a meta-analysis. This work is guided by previously published evidence maps whose focus was research priority setting [14–23] and the existing standards for conducting and reporting of systematic reviews of observational research [4, 24]. Evidence maps were created in Excel (Microsoft, Redmond, Washington, USA); however, it is possible to conduct the work using other database software.
Definition of design heterogeneity
In this paper, design heterogeneity refers to diversity across studies in sociodemographic and health characteristics of the populations studied; methods of study execution and data ascertainment; exposure (intervention), comparator and outcome definitions; and statistical approaches, as well as analyses conducted and reported.
Evidence-based mapping framework: steps for evaluating design heterogeneity
In order to present the evidence-based mapping framework in a way that other investigators can easily translate to their own research questions, the next section describes each step generally. The details of the application of the framework to a specific example are described in the subsequent section, including the systematic search process. This framework is designed to be dynamic. Although we recommend completing this work prior to finalizing a protocol and analysis plan for a meta-analysis, updating will be necessary when new data become available. We recommend that at least three investigators thoroughly use this framework: an author to abstract data at the onset and another to verify accuracy and a consensus of experts to review completed maps and evaluate aspects of design heterogeneity that may importantly influence meta-analysis of the selected studies.
Prior to applying this three-step framework, a PICO (participants, intervention (or exposure), comparator, and outcome) table is completed to identify key research components and to develop/clarify the research question . Once a group of studies have been selected for a meta-analysis, diversity is assessed across all included studies for each of the four PICO elements. An evaluation of confounding is added when including observational studies in the analysis. This framework can be used to document design heterogeneity across many study types, including randomized clinical trials.
Step 1: To assess diversity across selected studies for each participants, exposure/intervention, comparator, and outcome element
The goal of the first step of this framework is to evaluate the extent of diversity within each of the four PICO elements, although not in the same order. The framework begins with an assessment of the exposure variable (or intervention) across selected studies for two reasons: the exposure definition often is the driver of the analysis, and it is usually documented in the most detail in a publication.
For each study, the definition used for exposure is abstracted, including information on the measurement tools, timing of variable collection, and method/criteria (self-report, interviewer administered, medical or biochemical test). When possible, the exact language used to ascertain exposure status or details of the test performed is recorded. A diagram (evidence map) is created to describe how definitions of the study variable related to one another, quantitatively and qualitatively. The description of the variable definition, using the original language from the publication as much as possible, is summarized in a text box. Text boxes are organized to group together exposure variables with similar definitions. Similar definitions are physically grouped together in the diagram, and the review investigator assigns descriptive ‘category headings’ accordingly. The category of the exposure variable most frequently employed across studies is placed at the top of the diagram, with other categories arranged in order of decreasing frequency. An evaluation of whether the most frequently used definition is indeed the most appropriate definition was not undertaken at this point in the review process. The step should be examined later along with study quality and risk of bias.
The resulting map visually depicts patterns within the exposure definitions and is used to preliminarily evaluate whether the collective group of studies directly address the review question or address more than one distinct question. It also facilitates an initial assessment of frequently occurring subgroups of the exposure variable, which could be considered for stratified or sensitivity analyses in a meta-analysis.
In an etiologic example, the comparator is often the lowest exposure category and therefore included as part of the evaluation of the study variable. In analyses of randomized trials or nonrandomized studies, step 1 is repeated in order to evaluate diversity among definitions of the comparator across selected studies.
Step 1 is also repeated for the outcome variable (with particular attention to diagnostic method/criteria) and as needed for variables describing the study characteristics and population, examples include study location/ethnicity, gender, study size, study duration, timing of participant assessments, and baseline population characteristics such as age, body size, or health status. Univariate statistics (n, median, proportion, range) were used to describe the diversity of variable definitions (exposure, outcome, and co-variables) across included studies.
Step 2: To assess the design features and population characteristics associated with specific definitions of the exposure
The second step is to assess whether specific definitions of the exposure variable tended to aggregate with specific study design features or population characteristics. Using separate diagrams for each category of the exposure variable (identified in Step 1), important design features and population characteristics are listed for each study and qualitatively inspected to identify emerging patterns. Particular attention is focused on differences between categories of the exposure variable that are identified in Step 1 as potentially not directly answering the review question. Likewise, among exposure categories from studies directly answering the review question, the aggregation of study design/population characteristics is used to augment decisions from Step 1 about stratified/sensitivity analyses in a future review/meta-analysis.
Step 2 can be repeated as necessary to understand whether certain comparators or outcomes are associated with design or population characteristics.
Step 3: To evaluate the diversity in multivariable modeling strategies and assessment of confounding
The aim of step 3 is to evaluate co-variables selected for models by primary studies and to facilitate the selection of an adequately adjusted model or models for combining by meta-analysis of observational studies. Evidence-based mapping is used to visually display the patterns of co-variables adjusted for in each model as reported by each publication. A table summarizes how the exposure variable is analyzed in each study (for example, continuous measure or categories) and tallies the number of models from each publication and the number of covariates adjusted for in each model.
Every regression model is listed in sequential order as presented in the original research publication and a checklist format is used to summarize covariates adjusted for in each model. Covariates most frequently adjusted for in multivariable models across all studies are listed in the map header. A check denotes inclusion of a covariate in the model and a superscript is added to indicate the timing of the measurement of the covariate (for example, BL for baseline). Less frequently adjusted covariates are listed in a single column of the table.
For each multivariable model, the percent change in the exposure-outcome risk estimate from the age-adjusted model is calculated using the following equation:
(age-adjusted relative risk - multivariable adjusted relative risk)/(age-adjusted relative risk - 1).
This provides a quantitative assessment of the degree of fluctuation in the exposure-response risk estimate from the age-adjusted value, depending on the covariates included in a model.
Frequency (n), median, proportion, and range are used to describe across included studies the diversity in definitions of the study variable used for analysis (for example, how categories of exposure were defined), number of multivariable models presented, number of covariates adjusted for in multivariable models, and change in exposure-outcome risk estimate from the age-adjusted risk estimate.
Summarization of evidence-based mapping efforts
A single table, organized with sections for each PICO element, captures important findings and bridges the more practical need to concisely document and report design heterogeneity. We adopted a format that would be flexible for summarizing the large amounts of complex data organized by evidence maps. Using cohorts as the unit of analysis, for each major category of exposure (as determined by Step 1), the following were summarized: the distribution of important design and population characteristics (as determined in Step 2: n, percent), operationalization of the study variable in the multivariable model (Step 3: n, percent), the number of multivariable models presented in the original publication (Step 3: median, range), the number of covariates in multivariable models (Step 3: median, range), and the change in exposure-outcome risk estimate from the age-adjusted risk estimate (Step 3: median, range).
Illustration of method using prospective observational studies of sugar-sweetened beverages and type 2 diabetes
We illustrate the utility of an evidenced-based mapping framework using an example from nutritional epidemiology: sugar-sweetened beverages (SSB) and type 2 diabetes mellitus (T2D). This example is ideal for illustrating this framework, because studies of this relationship characteristically have considerable variability in study design.
First, we identified published work for the example. We used an electronic search strategy to identify all cohort studies of dietary sugar intake and T2D. Published research that met the following inclusion criteria were identified for full text review: 1) a prospective observational study (that is, dietary sugar consumption was measured in chronologic time prior to measurement of T2D) and 2) a study analyzing the risk of T2D associated with dietary sugar intake, dietary patterns, or glycemic load/index. To address the possibility that electronic search strategies might omit publications of findings not important enough (for example, null findings) for inclusion in the title, keywords, or abstract, our search ascertained published research on dietary patterns as well as dietary sugar intake. Additionally, we identified reviews and meta-analyses of epidemiologic studies on this topic in order to examine their reference lists.
We conducted database searches of PubMed and Scopus (inception to 10 March 2014). We limited our search of the PubMed database to human studies and English language publications, and used the following combination of search terms and medical subject headings (inception to 19 September 2013: 2,005 titles): sweetening agents, energy intake, calories, caloric intake, fructose, glucose, sucrose, monosaccharides, disaccharides, dietary carbohydrates, soda, sugar beverage, sweetened beverage, soft drink, dietary sugar, juice, sugar intake, sugary foods, sweets, sweet foods, carbohydrate intake, glycemic index, glycemic load, macronutrients AND diet, dietary patterns, dietary intake AND cohort studies, incidence, follow-up, prospective studies, meta-analysis AND Diabetes Mellitus, type 2 diabetes). We conducted a similar title, abstract, and keyword search of the Scopus database (1,143 titles): (diet* and sugar*) OR (diet* and pattern*) OR soda OR juice OR (sweet* and drink*) OR (sweet* and beverage*) OR (sweet* and food*) and (‘type 2 diabetes’). The search results were downloaded into Refworks (©Proquest 2012). Titles, abstracts, and keywords of all articles were examined, and those that continued to meet the inclusion criteria were ascertained for further full text review.
To ensure accurate identification of eligible studies, we conducted two pilot tests of our methodology prior to implementing the search described above. First, we assessed and revised a search strategy after retrieval and review of citations from several years, 2010 to 2013. The revised search strategy included more terms and more specific terms for dietary sugar, glycemic load/index and energy intake. This led to a broader, more inclusive search and the review of more titles. Second, two authors independently reviewed a subset of citations identified by our search strategy for eligibility (titles from 2012). Because both authors identified the same articles (inter-rater reliability = 100%), decisions regarding inclusion/exclusion reliably were based on a review by one author.
Identification and tracking of eligible publications
A flow chart tracked eligible publications identified by the literature searches and illustrated a two-stage evaluation process (Figure 1).
As part of the process of identifying eligible cohorts we displayed how epidemiologic studies of SSB fit into the broader field of research on dietary sugar intake and T2D. We tabulated the cohorts that published on measures of dietary sugar intake (including SSB) by study size. The number of publications and corresponding cohorts were depicted for each definition of dietary sugar intake, including sweetened beverages and macronutrients (sucrose, fructose, and glucose). Table 1 was based on the World Health Organization and the Food and Agricultural Organization of the United Nations definitions  of dietary sugar intake as ‘all monosaccharides and disaccharides added to foods by the manufacturer, cook, or consumer; sugars naturally present in honey, syrups and fruit juices .’ Table 1 facilitated identifying the studies that focus on SSB as a subset of all studies on dietary sugar intake.
For eligible prospective observational studies of SSB and T2D, we created detailed data abstraction tables. For each study, we abstracted data on sample size and population characteristics (for example, country, baseline age and body size), SSB definition and consumption at baseline, T2D diagnosis, dietary assessment timing and tools, duration of follow-up, timing of ascertainment of beverage consumption, variables included in multivariable models, and statistical analyses. We present this work without linking study design features to study results. We recommend this step in order to minimize as much as possible selection bias when planning a protocol for a subsequent meta-analysis. One author abstracted data at the onset, all authors contributed to strategy and map designs, and another author verified the accuracy of data abstracted at the end stage.
Participants, exposure/intervention, comparator, and outcome elements
The following PICO elements were specified for this work:
Participants/study design: adults from the general population without T2D/prospective observational studies.
Exposure: sugar-sweetened beverage consumption (our example is etiologic; therefore the ‘I’ in PICO is exposure).
Comparator: low or no consumption of sugar-sweetened beverages.
Outcome: incident T2D.
Step 1: Assessing diversity across selected studies for each participants, exposure/intervention, comparator, and outcome element
Evidence maps were used to categorize studies based on the definition of the exposure variable, outcome, and population characteristics. Exposure characterization took into account the type of beverage, data collection instruments, and frequency/timing of data collection (Figure 2). Variation in definitions of T2D was evaluated based on criteria for diagnosis and method of ascertainment such as by a physician or self-report. Using the study as the unit of analysis, univariate statistics (n, median, proportion, range) were used to describe across included cohorts heterogeneity of SSB intake (exposure and comparator), T2D diagnosis (outcome), and the following population/study characteristics: study location, gender, study size, duration of follow-up, baseline BMI, and baseline SSB consumption.
Step 2. Describing design features and population characteristics associated with the exposure across eligible cohorts
Cohorts were organized in a diagram according to category of the sweetened beverage consumption identified in Step 1. Design and population characteristics for cohorts falling into each beverage category were summarized. This provided an organized illustration of whether specific definitions of SSB tended to aggregate with specific study design features or population characteristics (Figure 3).
Step 3. Describing modeling strategies across eligible cohorts (assessing confounding)
Evidence-based mapping visually displayed the patterns of covariates adjusted for in each model of SSB and T2D as reported by each publication (Figure 4). A check denoted adjustment for covariates age, smoking, physical activity, family history, alcohol intake, diet quality score, energy intake, and body mass index. Overall and within important strata of the study variable, diversity of multivariable modeling strategies was described by summarizing operationalization of the SSB intake (n, percent), the maximum number of models of SSB and T2D presented in the original publication (median, range), the maximum number of covariates in models (median, range), and the maximum change in SSB-T2D risk estimate from the age-adjusted risk estimate (median, range).
Summarization of evidence-based mapping efforts (Table 2)
Using cohorts as the unit of analysis, for categories defined by SSB-intake (as determined by Step 1), the following was tabulated: the diversity of design and population characteristics, the study variable and the outcome (as determined in Steps 1 and 2: n, percent), operationalization of the SSB in the multivariable model (Step 3: n, percent), the number of multivariable models of SSB-T2D presented in the original publication (Step 3: median, range), the number of covariates in SSB-T2D multivariable models (Step 3: median, range), and the change in SSB-T2D risk estimate from the age-adjusted risk estimate (Step 3: median, range).
Literature search and identification of eligible publications
The search results are summarized in Figure 1. Briefly, a total of 3,148 titles were reviewed (2,005 from Pubmed and 1,143 from Scopus). After duplicate removal (N = 180), 2,968 titles were examined and reviewed in a two-step process. We identified 146 titles broadly on topic; a second review revealed that 90 were prospective epidemiology studies, commentary and reviews of sugar and T2D. Excluded were publications whose focus was not dietary sugar intake and T2D and those that were case–control, cross-sectional and ecologic studies (N = 56). Full-text review of the 90 publications identified 22 primary research publications of the relationship between dietary sugar intake and T2D [28–49] and one meta-analysis with previously unpublished data . A bibliography search of systematic reviews and meta-analyses did not identify any more potentially eligible titles [50–52].
Table 1 summarizes 21 publications from 17 cohorts that report the relationship between the following measures of dietary sugar intake and T2D [28–48]: category of SSB (broadly defined including some studies which include both artificially and sugar-sweetened beverages) and sugar-related macronutrients (total sugars, sucrose, fructose, and glucose). Studies of SSB (broadly defined) represent the majority of the published work on measures of dietary sugar intake, with 15 publications from 11 cohorts, most with more than 5,000 study participants. Nine publications from eight cohorts analyze sugar-related macronutrients and T2D, with total sugars and sucrose being the most frequently assessed. With the exception of a Finnish study initiated prior to 1970, all studies of macronutrients have >25,000 participants. A very small Swedish study assessing cake and biscuit consumption  was not summarized by Table 1.
Organizing and evaluating design heterogeneity among cohorts assessing sugar-sweetened beverages and type 2 diabetes
For the assessment of design heterogeneity, we selected one publication from each cohort that had multiple publications (n = 3 cohorts). We selected the one in which SSB was either the main study variable or the definition was the clearest. Details of unselected publications are noted at the bottom of Figure 1.
Step 1. Assessing diversity across selected studies for each participants, exposure/intervention, comparator, and outcome element
Study variables and outcomes were categorized into logical groups by definitions reported in each of the 11 eligible cohorts. No two cohorts define the main study variable alike. As shown in Figure 2, two broad definitions of sweetened beverage consumption emerged: 1) three studies used the nonspecific definition soft drinks (SD) that included both sugar and artificially-sweetened beverages, and 2) eight studies restricted the definition to SSB only. Three distinct subgroups were identified among cohorts defining the study variable as exclusively sugar-sweetened. The general definition ‘SSB, not 100% juice’ includes all drinks with added sugar (sodas, colas, other carbonated SSB, and noncarbonated SSBs such as fruit punches, lemonades or other fruit drinks). Two other SSB patterns were identified within the remaining five cohorts based on whether they excluded beverages (SSB minus, three cohorts) or included additional beverages (SSB plus, two cohorts) from the anchor definition (SSB, not 100% juice). We found that investigators more frequently excluded beverages from the anchor definition, most broadly noncarbonated soft drinks as an entire group or fruit drinks. Two studies added beverages to the definition, one orange and grapefruit juice and the other non-alcoholic beer. This detailed characterization of the study variable identified two broad research questions addressed by this series of selected studies: T2D risk associated with intake of 1) SSB only or 2) any SD (artificially or sugar-sweetened).
Of the 11 cohorts, method of diagnosis was based on self-report (n = 6; 3 of the 6 were studies of health professionals), registry linkage (n = 2), and an examination by a health professional (n = 3).
Univariate analysis of design features and population characteristics across the 11 cohorts revealed heterogeneity in study location (US: n = 6, Europe: n = 2, China/Japan: n = 3), size (>5000 cases: n = 2, 500 to 4999 cases: n = 6, 1 to 499 cases: n = 3), duration of follow-up (<10 years: n = 4, 10 to 14 years: n = 4, 15 or more years: n = 3); mean baseline body mass index (<24 kg/m2: n = 3, 24 to 26 kg/m2: n = 4, > 26 kg/m2: n = 4), ascertainment of diet (food frequency questionnaire: n = 8, diet history: n = 3), and frequency of diet assessment (baseline only: n = 7, twice every 6 years: n = 1, every 4 years: n = 3). We also found a relatively low consumption of SSB across the 11 cohorts with nearly half (n = 5) reporting that 10% or fewer participants consumed one or more servings per day.
Step 2. Describing design features and population characteristics associated with the study variable across eligible cohorts
The upper portion of Figure 3 compares design features and population characteristics of cohorts defining sweetened beverage consumption as SSB only and SD. SSB cohorts were mainly US-based (n = 6), completed diet assessments at least twice (n = 4), followed subjects for fewer than 10 years (n = 4), and reported a mean body mass index (BMI) ≥24 kg/m2 (n = 7). We identified three levels of SSB consumption ascertained at the baseline visit of these eight combined cohorts: frequency of one or more servings a day (highest consumption group for each study) was reported by 10% or fewer (n = 2), between 11 and 15% (n = 3) and more than 15% (n = 2) of cohort participants.
Design and population characteristics for SD cohorts presented differently. Two of three SD cohorts were Asian populations; the one western population was a small Finnish study that enrolled participants between 1967 and 1972. Mean BMI was <24 kg/m2 in two of the three studies, SD consumption was measured only at the baseline visit (10 to 12 years prior to maximum follow-up duration), and SD consumption overall for the three cohorts was low. Comparison of the study design features of cohorts assessing SSB only with SD suggests it would not be sensible to combine all eleven studies in a meta-analysis; instead the main pooled analysis should include the eight SSB-only studies.Further division of SSB cohorts into categories of SSB, SSB minus, and SSB plus uncovered patterns according to study design, as shown in the lower portion of Figure 3. Large studies of women (Black Women’s Health Study (BWHS), Nurse’s Health Study (NHS), NHSII) with multiple dietary assessments more narrowly defined SSB consumption as excluding noncarbonated drinks (SSB, minus). The multicultural cohorts initiated to study atherosclerosis (Multi-Ethnic Study of Atherosclerosis (MESA), Atherosclerosis Risk in Community study (ARIC)) more broadly defined SSB by including either juice or non-alcoholic beer. These cohorts have higher baseline SSB consumption when compared to studies defining SSB more narrowly. No clear pattern emerged for cohorts defining the study variable as SSB, not 100% juice. Stratification of a meta-analysis on SSB subcategories and gender may additionally be important for understanding pooled T2D risk estimates.
We used a similar process to evaluate design heterogeneity of the outcome definitions used in these cohorts. While several different criteria for T2D were used in the 11 cohorts, the main defining characteristic was whether the diagnosis was based on self-report (all included a validation study), physical examination, or linkage to a registry or other health database. With the exception of the European Investigation of Cancer (EPIC) study, which verified cases via a registry, the larger studies (>25,000 participants) relied on self-reported diagnoses. Three studies conducted routine physical exams.
Step 3. Diversity of modeling strategies (confounding)
Multivariable models compared risk in the highest category of consumption (quartile or quintile) to the lowest. Figure 4 summarizes the different definitions of high and low categories of sweetened beverage consumption. Among studies of SSB only, the highest consumption category was 1+ glasses or servings each day in 6 (of 8) cohorts and 2+ drinks or cups each day for 2 (serving sizes varied). In comparison, the highest consumption category for two of three cohorts evaluating SD was less than one serving per day. Never or rare sweetened beverage consumption was the most frequently employed reference group (7 of 11), followed by never consumption (3 of 11). ARIC was the only study to include more frequent consumption in the reference group: up to one cup of SSB per day.Figure 4 visually depicts multivariable models and the pattern of covariate adjustment across 11 cohorts. The majority of cohorts present a multivariate model adjusting for age, physical activity, smoking, family history, alcohol intake, energy intake and BMI. Four studies adjust for a diet quality score, although all measure and adjust for some aspect of diet. Many models further adjust for multiple other covariates (up to 17).
Many models use different definitions to adjust for the listed co-variables and 9 of 11 adjust for covariates as measured at baseline. For example, measures of other dietary factors ranged from one variable measuring dietary fiber to healthy eating scores based on the entire diet (for 3 cohorts only). Likewise body mass index is adjusted in many ways: as a continuous variable (3 of 11 cohorts), a categorical variable (5 of 11 cohorts), and as measured at baseline (6 of 11 cohorts).Multivariable models adjust for between 5 and 17 covariates which corresponded to a 46% to 95% maximum reduction from the age-adjusted model in T2D risk associated with SSB-only intake (Figure 4). Change in the risk estimate was most pronounced among the large cohorts of US women using sugar-sweetened carbonated beverages as the study variable: reductions were as large as 95% in the BWHS, 61% in the NHS, and 67% in the NHSII. Change in risk estimates with addition of covariates was less pronounced among SD cohorts (range, 2-26%).
Summarization of evidence-based mapping efforts
Table 2 concisely summarizes the considerable amount of variability in study design, population characteristics, and statistical analysis among the 11 cohorts of sweetened beverages and T2D. This table represents a proto-type for a universal table on study design heterogeneity summarizing key design features uncovered by detailed evidence-based mapping efforts and organized according to PICO elements. The results in Table 2 are presented stratified by SSB only and SD to display the association of different definitions of the study variable with specific design and population characteristics. In addition, Table 2 highlights diversity in statistical analysis and provides a benchmark for the potential for confounding overall and for the two primary definitions of sweetened beverage consumption.
Evidence-based mapping can be used as a tool to improve the assessment and reporting of design heterogeneity prior to meta-analysis of epidemiologic studies. The framework described herein is useful for all study designs, but particularly for observational epidemiologic studies, which are complex and rich in important detail. If studies are found to be similar enough to combine via meta-analysis, this framework is useful for evaluating diversity in study designs, particularly statistical methods; facilitating the logical categorization of studies for stratified and sensitivity analyses when designing a protocol or analysis plan for a meta-analysis; and developing tools for model selection in meta-analysis of observational studies reporting multiple multivariable models. A standard table for summarizing the results from the 3 steps in this framework is essential for displaying the multi-dimensionality of diversity across a group of selected studies and to aid interpretation of a pooled risk estimate.
Evidence maps are ideal tools for characterizing heterogeneity prior to a meta-analysis. Previously they have been used for research priority setting by the Cochrane Collaboration  and other organizations such as the Agency for Healthcare Research and Quality [54–56]. In addition to organizing a complex body of research, another defining feature of evidence mapping is that the mapping of study characteristics is undertaken without linking to study results . Although we used prospective observational studies of SSB and T2D to explain our approach, the framework is robust and this strategy can be applied to other exposure-disease relationships and epidemiologic study types.
The evidence-based mapping strategies using SSB and T2D as an example facilitated the logical grouping of studies on key design features and suggested subgroups of studies appropriate for statistical summation via meta-analysis. For example, we found considerable variability in the definition and methods of collection of the exposure variable (sweetened beverages). Most notable is the inclusion of artificially-sweetened beverages in the definition in 3 of 11 cohorts. Consequently two broad research questions are addressed by this series of selected studies: T2D risk associated with intake of 1) SSB only and 2) any soft drink (artificially or sugar-sweetened). Diversity across studies in the definition of the exposure variable may be due to a combination of factors, including availability of data from the dietary assessment tool, the definition used by the study investigators, or the level of detail provided in the publication. Improving the interpretability of meta-analyses will require investigators of primary studies, in as much as possible, to define variables, conduct analyses, and report findings with an eye towards how their results may be compared to or possibly combined with other studies in the future.
The systematic approach described herein culminated in a prototype for a table that can be employed widely for reporting the extent and multi-dimensional nature of design heterogeneity across eligible studies in a meta-analysis. This table is recommended in addition to the classic table 1 in a systematic review (which usually describes studies individually). A standard table summarizing design heterogeneity across all selected studies will bring to the fore many elements necessary for interpreting the pooled risk estimate from a meta-analysis. One of many examples from our assessment of cohorts of SSB and T2D is 7 of 11 studies measured beverage intake only once at baseline, each using a different diet questionnaire and following participants for T2D from 7 to 16 years. The etiologically relevant time period for most chronic diseases, including T2D, is most often not known, and a one-time measurement of dietary intake may not capture intake in the relevant time frame. This is a fundamental consideration when interpreting results of chronic disease studies, including meta-analysis of these data.
To our knowledge, this may be the first detailed report of diversity of statistical modeling approaches among observational studies selected for a meta-analysis. A common practice for reporting modeling strategies in meta-analyses of observational studies is to provide a list of included covariates by study. We suggest summarizing the following across selected studies: the number of multivariable models presented, the number of covariates adjusted for in multivariable models, and the fluctuation in the fully adjusted risk estimate relative to the age-adjusted or most minimally adjusted model. As an example, it may add confidence about the results of a meta-analysis combining models that all adjust for the same 5 variables and with little fluctuation in the most fully adjusted risk estimate compared to the age or most minimally adjusted model. In contrast, cohorts of SSB and T2D reported up to 8 multivariable models adjusting for between 11 and 17 covariates. The results revealed a 2-95% reduction in risk of T2D associated with sweetened beverage consumption in fully adjusted relative to minimally/age- adjusted models (Table 2). The latter finding was most pronounced among the eight studies of SSB only, where adjustment for between 11 to 17 covariates resulted in a 46 to 95% reduction in the SSB-T2D risk estimate (Table 2). In other words, in many studies adjustment for covariates explained half to all of the association between SSB and T2D and should be considered when analyzing and interpreting a meta-analysis of the data.
Selection of statistical models by the study investigators from the primary publication and by a review investigator for a meta-analysis also influences the final outcome of a pooled analysis of observational studies. This particular bias, called selective analysis reporting, has recently been discussed as a major concern for meta-analysis of nonrandomized studies, but also applies to observational etiologic investigations . Covariate selection and modeling strategies require careful consideration in the final interpretation of a pooled analysis of SSB and T2D; a single estimate for a pooled risk may be an oversimplification of complex data. Evidence maps can help facilitate the selection of additional models for sensitivity analyses in a meta-analysis.
Heterogeneity of studies can be a reason not to perform a meta-analysis. For example, a systematic review of whole grain foods and T2D that had intended to complete a meta-analysis concluded a qualitative synthesis was more appropriate for the data . Other investigators have determined that a meta-analysis of this topic was informative [59, 60]. A tool (such as the proposed summary table) that clearly displays design heterogeneity may be helpful in weighing both sides of this type of debate.
Systematic reviews and meta-analyses are well accepted research synthesis methods that serve to inform researchers, policy makers and, increasingly, the public of the potential causes of disease and the extent to which disease (or preventive) interventions are effective. The efficiency of these efforts depends largely on the quality of data from primary studies and a clear assessment of the extent to which that data can be combined.
We illustrate a framework employing evidenced-based mapping to organize, evaluate and document design heterogeneity. This exercise culminated in a recommendation for a standardized table format that clearly summarizes design heterogeneity of eligible studies, with the goal of informing a protocol for meta-analysis and subsequently facilitating interpretation of summary risk estimates after quantitative synthesis. We recommend expanding the practice of meta-analysis of cohort studies to include a standard table that summarizes design heterogeneity. Addition of this table to reporting of meta-analyses provides the reader with more evidence to interpret the summary risk estimates.
MDA and DLW are independent consultants. MDA works for (and owns) EpiContext. DLW works for (and owns) DLW Consulting Services. CLF is a faculty member at George Mason University. This work was undertaken with the contractual understanding that the funder had no control over the design, analysis, and interpretation of the study and its results.
Atherosclerosis Risk in Community Study
body mass index
Black Women’s Health Study
European Investigation of Cancer
Multi-Ethnic Study of Atherosclerosis
Nurse’s Health Study
participants, exposure/intervention, comparator, and outcome
type 2 diabetes.
Borenstein M, Hedges LV, Higgins JPT, Rothstein HR: Introduction to Meta-analysis. 2008, Hoboken, NJ: Wiley
Egger M, Davey Smith G, Schneider M: Systematic reviews of observational studies. Systematic Reviews in Health Care: Meta-analysis in Context. Edited by: Egger M, Davey Smith G, Altman DG. 2001, London: BMJ Books, 211-227. 2
Wong O, Raabe GK: Application of meta-analysis in reviewing occupational cohort studies. Occup Environ Med. 1996, 53: 793-800.
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB: Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000, 283: 2008-2012.
Blettner M, Sauerbrei W, Schlehofer B, Scheuchenpflug T, Friedenreich C: Traditional reviews, meta-analyses and pooled analyses in epidemiology. Int J Epidemiol. 1999, 28: 1-9.
Mosteller F, Colditz GA: Understanding research synthesis (meta-analysis). Annu Rev Pub Health. 1996, 17: 1-23.
Thompson SG: Why and how sources of heterogeneity should be investigated. Systematic Reviews in Health Care: Meta-analysis in Context. Edited by: Egger M, Davey Smith G, Altman DG. 2001, London: BMJ Books, 157-174. 2
Thompson SG: Why sources of heterogeneity in meta-analysis should be investigated. BMJ. 1994, 309: 1351-1355.
Deeks JJ, Higgins JPT, Altman DG: Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. 2008, Chichester England: Wiley-Blackwell, 243-296.
Higgins JP, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. BMJ. 2003, 327: 557-560.
McElvenny DM, Armstrong BG, Järup L, Higgins JP: Meta-analysis in occupational epidemiology: a review of practice. Occup Med. 2004, 54: 336-344.
Reeves BC, Higgins JPT, Ramsay C, Shea B, Tugwell P, Wells GA: An introduction to methodological issues when including non-randomised studies in systematic reviews on the effects of interventions. Res Syn Meth. 2013, 4: 1-11.
Hannes K, Booth A, Harris J, Noyes J: Celebrating methodological challenges and changes: reflecting on the emergence and importance of the role of qualitative evidence in Cochrane reviews. Syst Rev. 2013, 2: 84-
Althuis MD, Weed DL: Evidence mapping: methodologic foundations and application to intervention and observational research on sugar-sweetened beverages and health outcomes. Am J Clin Nutr. 2013, 98: 755-768.
Callahan P, Liu P, Purcell R, Parker AG, Hetrick SE: Evidence map of prevention and treatment interventions for depression in young people. Depress Res Treat. 2012, 2012: 820735-
Gough D, Thomas J, Oliver S: Clarifying differences between review designs and methods. Syst Rev. 2012, 1: 28-
Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL: The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011, 17: 11-92.
Hetrick SE, Parker AG, Callahan P, Purcell R: Evidence mapping: illustrating an emerging methodology to improve evidence-based practice in youth mental health. J Eval Clin Pract. 2010, 16: 1025-1030.
Levac D, Colquhoun H, O'Brien KK: Scoping studies: advancing the methodology. Implement Sci. 2010, 5: 69-
Liu Y, Wen XM, Lui EL, Friedman SL, Cui W, Ho NP, Li L, Ye T, Fan ST, Zhang H: An evidence map of interventions across premorbid, ultra-high risk and first episode phases of psychosis. Schizophr Res. 2010, 123: 37-44.
Curran C, Burchardt T, Knapp M, McDaid D, Li B: Challenges in multidisciplinary systematic reviewing: a study on social exclusion and mental health policy. Soc Pol Admin. 2007, 41: 289-312.
Arksey H, O’Malley L: Scoping studies: towards a methodological framework. Int J Social Research Methodology. 2005, 8: 19-32.
Katz DL, Williams AL, Girard C, Goodman J, Comerford B, Behrman A, Bracken MB: The evidence base for complementary and alternative medicine: methods of evidence mapping with application to CAM. Altern Ther Health Med. 2003, 9: 22-30.
Moher D, Liberati A, Tetzlaff J, Altman DG: PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009, 339: b2535-
de Costa Santos CM, de Mattos Pimenta CA, Nobre MR: The PICO strategy for the research question construction and evidence search. Rev Lat Am Enfermagem. 2007, 15: 508-511.
Mann J, Cummings JH, Englyst HN, Key T, Liu S, Riccardi G, Summerbell C, Uauy R, van Dam RM, Venn B, Vorster HH, Wiseman M: FAO/WHO scientific update on carbohydrates in human nutrition: conclusions. Eur J Clin Nutr. 2007, 61 (Suppl 1): S132-S137.
Te Morenga L, Mallard S, Mann J: Dietary sugars and body weight: systematic review and meta-analyses of randomised controlled trials and cohort studies. BMJ. 2012, 346: e7492-
Palmer JR, Boggs DA, Krishnan S, Hu FB, Singer M, Rosenberg L: Sugar-sweetened beverages and incidence of type 2 diabetes mellitus in African American women. Arch Intern Med. 2008, 168: 1487-1492.
Consortium IA: Consumption of sweet beverages and type 2 diabetes incidence in European adults: Results from EPIC-InterAct. Diabetologia. 2013, 56: 1520-1530.
Sluijs I, Beulens JW, van der Schouw YT, van der A DL, Buckland G, Kuijsten A, Schulze MB, Amiano P, Ardanaz E, Balkau B, Boeing H, Gavrila D, Grote VA, Key TJ, Li K, Nilsson P, Overvad K, Palli D, Panico S, Quirós JR, Rolandsson O, Roswall N, Sacerdote C, Sánchez MJ, Sieri S, Slimani N, Spijkerman AM, Tjønneland A, Tumino R, Sharp SJ, InterAct consortium: Dietary glycemic index, glycemic load, and digestible carbohydrate intake are not associated with risk of type 2 diabetes in eight European countries. J Nutr. 2013, 143: 93-99.
Fagherazzi G, Vilier A, Sartorelli DS, Lajous M, Balkau B, Clavel-Chapelon F: Consumption of artificially and sugar-sweetened beverages and incident type 2 diabetes in the etude epidé miologique auprè s des femmes de la mutuelle Générale de l'education nationale-european prospective investigation into cancer and nutrition cohort. Am J Clin Nutr. 2013, 97: 517-523.
Sluijs I, van der Schouw YT, van der A DL, Spijkerman AM, Hu FB, Grobbee DE, Beulens JW: Carbohydrate quantity and quality and risk of type 2 diabetes in the european prospective investigation into cancer and nutrition-Netherlands (EPIC-NL) study. Am J Clin Nutr. 2010, 92: 905-911.
Schulze MB, Schulz M, Heidemann C, Schienkiewitz A, Hoffmann K, Boeing H: Carbohydrate intake and incidence of type 2 diabetes in the European prospective investigation into cancer and nutrition (EPIC)-Potsdam study. Br J Nut. 2008, 99: 1107-1116.
de Koning L, Malik VS, Rimm EB, Willett WC, Hu FB: Sugar-sweetened and artificially sweetened beverage consumption and risk of type 2 diabetes in men. Am J Clin Nutr. 2011, 93: 1321-1327.
Bhupathiraju SN, Pan A, Malik VS, Manson JE, Willett WC, van Dam RM, Hu FB: Caffeinated and caffeine-free beverages and risk of type 2 diabetes. Am J Clin Nutr. 2013, 97: 155-166.
Meyer KA, Kushi LH, Jacobs DR, Slavin J, Sellers TA, Folsom AR: Carbohydrates, dietary fiber, and incident type 2 diabetes in older women. Am J Clin Nutr. 2000, 71: 921-930.
Eshak ES, Iso H, Mizoue T, Inoue M, Noda M, Tsugane S: Soft drink, 100% fruit juice, and vegetable juice intakes and risk of diabetes mellitus. Clin Nutr. 2013, 32: 300-308.
Hodge AM, English DR, O'Dea K, Giles GG: Glycemic index and dietary fiber and the risk of type 2 diabetes. Diabetes Care. 2004, 27: 2701-2706.
Bazzano LA, Li TY, Joshipura KJ, Hu FB: Intake of fruit, vegetables, and fruit juices and risk of diabetes in women. Diabetes Care. 2008, 31: 1311-1317.
Colditz GA, Manson JE, Stampfer MJ, Rosner B, Willett WC, Speizer FE: Diet and risk of clinical diabetes in women. Am J Clin Nutr. 1992, 55: 1018-1023.
Schulze MB, Manson JE, Ludwig DS, Colditz GA, Stampfer MJ, Willett WC, Hu FB: Sugar-sweetened beverages, weight gain, and incidence of type 2 diabetes in young and middle-aged women. JAMA. 2004, 292: 927-934.
Odegaard AO, Koh WP, Arakawa K, Yu MC, Pereira MA: Soft drink and juice consumption and risk of physician-diagnosed incident type 2 diabetes: The Singapore Chinese health study. Am J Epidemiol. 2010, 171: 701-708.
Janket SJ, Manson JE, Sesso H, Buring JE, Liu S: A prospective study of sugar intake and risk of type 2 diabetes in women. Diabetes Care. 2003, 26: 1008-1015.
Paynter NP, Yeh HC, Voutilainen S, Schmidt MI, Heiss G, Folsom AR, Brancati FL, Kao WH: Coffee and sweetened beverage consumption and the risk of type 2 diabetes mellitus: The atherosclerosis risk in communities study. Am J Epidemiol. 2006, 164: 1075-1084.
Nettleton JA, Lutsey PL, Wang Y, Lima JA, Michos ED, Jacobs DR: Diet soda intake and risk of incident metabolic syndrome and type 2 diabetes in the multi-ethnic study of atherosclerosis (MESA). Diabetes Care. 2009, 32: 688-694.
Ahmadi-Abhari S, Luben RN, Powell N, Bhaniani A, Chowdhury R, Wareham NJ, Forouhi NG, Khaw KT: Dietary intake of carbohydrates and risk of type 2 diabetes: The european prospective investigation into cancer-Norfolk study. Br J Nutr. 2014, 111: 342-352.
Montonen J, Jarvinen R, Knekt P, Heliovaara M, Reunanen A: Consumption of sweetened beverages and intakes of fructose and glucose predict type 2 diabetes occurrence. J Nutr. 2007, 137: 1447-1454.
Sakurai M, Nakamura K, Miura K, Takamura T, Yoshita K, Nagasawa SY, Morikawa Y, Ishizaki M, Kido T, Naruse Y, Suwazono Y, Sasaki S, Nakagawa H: Sugar-sweetened beverage and diet soda consumption and the 7-year risk for type 2 diabetes mellitus in middle-aged Japanese men. Eur J Nutr. 2014, 53: 1137-1138.
Lundgren H, Bengtsson C, Blohmé G, Isaksson B, Lapidus L, Lenner RA, Saaek A, Winther E: Dietary habits and incidence of noninsulin-dependent diabetes mellitus in a population study of women in Gothenburg, Sweden. Am J Clin Nutr. 1989, 49: 708-712.
Malik VS, Popkin BM, Bray GA, Després JP, Willett WC, Hu FB: Sugar-sweetened beverages and risk of metabolic syndrome and type 2 diabetes: a meta-analysis. Diabetes Care. 2010, 33: 2477-2483.
Alhazmi A, Stojanovski E, McEvoy M, Garg ML: Macronutrient intakes and development of type 2 diabetes: a systematic review and meta-analysis of cohort studies. J Am Coll Nutr. 2012, 31: 243-258.
Hauner H, Bechthold A, Boeing H, Brönstrup A, Buyken A, Leschik-Bonnet E, Linseisen J, Schulze M, Strohm D, Wolfram G, German Nutrition Society: Evidence-based guideline of the German Nutrition Society: carbohydrate intake and prevention of nutrition-related diseases. Ann Nutr Metab. 2012, 60 (Suppl 1): 1-58.
Jaramillo A, Welch VA, Ueffing E, Gruen RL, Bragge P, Lyddiatt A, Tugwell P: Prevention and self-management interventions are top priorities for osteoarthritis systematic reviews. J Clin Epidemiol. 2013, 66: 503-510.
Greer N, Brasure M, Wilt TJ: Wheeled mobility (wheelchair) service delivery. [http://www.ncbi.nlm.nih.gov/books/NBK83951/]
Wysocki A, Butler M, Shamliyan T, Kane RL: Whole-body vibration therapy for osteoporosis. [http://www.ncbi.nlm.nih.gov/books/NBK83043/]
Chung M, Dahabreh IJ, Hadar N, Ratichek SJ, Gaylor JM, Trikalinos TA, Lau J: Emerging MRI technologies for imaging musculoskeletal disorders under loading stress. [http://www.ncbi.nlm.nih.gov/books/NBK82287/]
Norris SL, Moher D, Reeves BC, Shea B, Loke Y, Garner S, Anderson L, Tugwell P, Wells G: Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions. Res Syn Meth. 2013, 4: 36-47.
Priebe MG, van Binsbergen JJ, de Vos R, Vonk RJ: Whole grain foods for the prevention of type 2 diabetes mellitus. Cochrane Database Syst Rev. 2008, 1: CD006061
Ye EQ, Chacko SA, Chou EL, Kugizaki M, Liu S: Greater whole-grain intake is associated with lower risk of type 2 diabetes, cardiovascular disease, and weight gain. J Nutr. 2012, 142: 1304-1313.
de Munter JS, Hu FB, Spiegelman D, Franz M, van Dam RM: Whole grain, bran, and germ intake and risk of type 2 diabetes: a prospective cohort study and systematic review. PLoS Med. 2007, 4: e261-
This methodological study was supported by The Coca-Cola Company. By contractual agreement, decisions regarding the content of the manuscript − including the design, analysis and interpretation − rest solely with the authors.
All authors (MDA, DLW, CLF) contributed to the concept, design, interpretation of data and writing of the manuscript. MDA and CLF completed data abstraction and quality control. MDA created the evidence maps and tables. All authors (MDA, DLW, CLF) read and approved the final manuscript.
About this article
Cite this article
Althuis, M.D., Weed, D.L. & Frankenfeld, C.L. Evidence-based mapping of design heterogeneity prior to meta-analysis: a systematic review and evidence synthesis. Syst Rev 3, 80 (2014). https://doi.org/10.1186/2046-4053-3-80
- Evidence map
- Systematic review
- Sugar-sweetened beverages
- Type 2 diabetes