O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions
Systematic Reviews volume 6, Article number: 179 (2017)
In this comment, we summarize several scientific concerns with the recently published systematic review from O’Connor and colleagues that examined the relationship between proximity to animal-feeding operations and health of individuals in nearby communities. The authors utilized a bias tool not designed for environmental health research, erroneously excluded important studies, and incorrectly interpreted others. As a result, the conclusions drawn in the review misrepresent the evidence from the published literature, limiting its value to policymakers, researchers, and the public.
There is increasing recognition of the value of systematic reviews in the context of decision-making. These reviews are a popular tool in evidence-based medicine , and potential opportunities for their application in judging environmental health risks are actively discussed [2,3,4,5]. When properly conducted, systematic reviews can serve as critical inputs to the policy process and result in choices about interventions that are in the public’s interest and based on the best available science. Conversely, the use of systematic reviews that suffer from critical design flaws or lack rigorous evaluation of the evidence may hurt public health. We have concern that the recently published systematic review by O’Connor and colleagues  on the associations between living near an animal-feeding operation (AFO) and human health falls into the latter category. Although the review integrates some best practice guidelines for conducting systematic reviews, such as developing and registering a protocol beforehand in PROSPERO, the review appears substantially flawed in its approach and conduct; the authors utilized a bias tool that is inappropriate for environmental health research, erroneously excluded important studies, and incorrectly interpreted others. As a result, the ultimate conclusions of the review misrepresent evidence from the published literature. Given that the authors describe their review as having utility for decision-making and prioritizing research, a formal critique of the work is warranted to help calibrate its value for policymakers, researchers, and the public.
The systematic review suffers from shortcomings related to the application of the ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) tool. The tool was originally developed by the Cochrane Collaboration to evaluate the internal validity of studies of potentially beneficial medical interventions . Direct application to environmental health research is inappropriate because the evidence base and decision-making context differs substantially from clinical settings. For instance, many of the signaling questions to evaluate the study’s internal validity that inquire about intervention-specific details (adherence to the assigned intervention regimen, post-intervention variables influencing study participation, etc.) are not directly relevant to studies of potentially harmful environmental exposures. The use of the ROBINS-I tool in this context will inherently bias conclusions by finding that all included studies are of low quality, given the unavoidable limitations even in well-designed observational epidemiology studies.
Since 2009, there have been ongoing efforts to translate systematic review methods from the clinical sciences to the environmental health decision context, including the development of risk of bias tools in academic and government settings. There are now at least two published methods [2, 8] and several case studies [3, 5, 9,10,11,12,13] that illustrate approaches more directly applicable to review questions in the environmental health decision-making context than ROBINS-I.
The article-screening process and the application of the exclusion criteria were flawed in design and conduct. In the Level 2 screening, the authors specified that articles that did not include “more than one unit of measurement of exposure” would not be considered further in the review. There is no clear definition of this exclusion criterion offered in the text or justification for why the authors chose to exclude such studies. This criterion was used to eliminate eight studies identified in the Level 1 screening from further consideration. Inclusion of these articles would have increased the number of included studies by 50%, indicating that the impact of such an unjustified exclusion may be so strong in this case as to have led to spurious conclusions.
Beyond design flaws, we noted that numerous key papers examining infectious disease and respiratory outcomes related to community exposures to AFOs [14,15,16,17,18] were identified during the Level 1 screening but inappropriately excluded during the Level 2 screening. The rationale presented for excluding these papers was reported in the supplemental material as “the unit of analysis was not at the individual human level or the study looked at occupational exposure only”; however, neither of these statements applied to any of these studies. For example, Casey et al.’s analyses drew on individual-level longitudinal electronic health record data from primary care patients of the Geisinger Health System in Pennsylvania . This study was the first in the USA to demonstrate an association between residential proximity to swine AFOs and manure-applied crop fields and methicillin-resistant Staphylococcus aureus (MRSA) infection in a general population sample. The inappropriate exclusion of studies highlights serious concerns related to the rigor and consistency of analysis at the second level of screening and calls into question the validity of conclusions drawn by the authors on the basis of their review. In addition, we note that at least one of our papers  that appears relevant to the primary research question was not identified during the Level 1 screening process. Problems with the search strategy may suggest that other key publications could have also been missed, further limiting confidence in the authors’ conclusions.
The authors eliminated ecological studies from their review a priori. A better option may have been to include them while acknowledging the weaknesses and strengths; while inferences about individual-level effects of exposures cannot be inferred from ecologic analyses, these studies have utility in that they establish preliminary evidence of potential associations between exposure and health outcomes, often over large geographies. The authors also limited the scope of their review to the health effects in the community rather than including occupational exposures, despite the notion that workers often reside in communities with their families. While we do not take issue with the exclusion of occupational studies from the systematic review, we note that the environmental health literature offers many examples (e.g., lead, chromium, beryllium) of instances where early warnings arose from higher-level exposures seen in occupational settings . In the case of AFOs, a growing literature exists documenting relationships between occupational exposures and infectious disease and respiratory outcomes [21,22,23,24,25,26,27].
Multiple errors in study interpretation and presentation were also noted. For example, the authors stated that Feingold et al.  reported an association between MRSA carriage and livestock density, but the Feingold study utilized a case-case analysis comparing MRSA ST-398 to other MRSA strains, not to a control population that would allow such a conclusion to be drawn. Other examples include the erroneous presentation of data from the Avery et al.  study, which O’Connor et al. mistakenly list in Table 1 as part of the Community Health Effects of Industrial Hog Operations (CHEIHO) study, rather than as part of a smaller pilot investigation. Also contrary to its presentation in Table 1, Schinasi et al.  employed multivariate fixed effects regression models as the primary analytic method.
O’Connor et al.’s evaluation of risk of bias across studies may reflect a misunderstanding of the design and analysis used in some of the papers reported. The authors state, “[b]ecause the studies were cross-sectional in nature, adjustment for confounders would not provide protection against residual confounding.” We disagree with these assertions for two reasons. First, methods exist to evaluate the degree of bias introduced from unmeasured confounders  and this residual confounding does not always prevent causal inference from cross-sectional studies . Second, multiple studies included in the review were longitudinal in nature [30, 33, 34]. For example, the CHEIHO study used a case-crossover design and adjusted for within-person, time-invariant confounders. As a result, the models quantified the impact of exposure variability on response variability within each person and, in so doing, adjusted for all stable, potentially confounding covariates that varied between people . While it is true that unmeasured time-varying confounding may remain, these models accounted for much of the confounding that is normally a concern .
In their discussion section, O’Connor and colleagues call into question studies that are older than 10 years (e.g., [30, 33, 34, 37, 38]). The authors suggest that “dramatic” changes in environmental regulation, animal housing, and manure management may have occurred that would invalidate these studies. However, no evidence supporting this point is presented. Additionally, there is no reason to believe that biologic effects of exposures related to AFOs would change as a function of time.
Policymakers depend on peer-reviewed scientific research as a credible source of information upon which decisions can be built. As a recent example, some of the studies of community risks of AFOs that were excluded in this review [14, 15] and other related research syntheses  played an important role in advocacy efforts (e.g., [40, 41]) around two state policies aimed at eliminating non-therapeutic antibiotic use in animal agriculture [42, 43]. Despite the utility of these studies, keeping up with and interpreting the scientific literature is beyond the capacity of most policymakers; thus, the conclusions of systematic reviews can be instrumental in supporting the development of evidence-based policy. When design flaws and less-than-rigorous conduct influence their results, however, the findings of these reviews may be used to adversely impact the policy process or resultant interventions. In contrast to the findings of the authors of this review, we believe there is a considerable and growing body of rigorously conducted scientific evidence that suggests connections between living near AFOs and adverse health outcomes . With a different tool to evaluate the evidence, and a more rigorous application of that tool, it is entirely possible the authors would have come to a different conclusion regarding the community health risks of AFOs.
Ioannidis J. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514.
Woodruff TJ, Sutton P. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect. 2014;122:1007.
Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122
Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, Guyatt G, Hooijmans C, Langendam M, Mandrioli D. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92:611–6.
Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, Ratcliffe JM, Kraft AD, Schünemann HJ, Schwingl P, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92–93:617–29.
O’connor AM, Auvermann BW, Dzikamunhenga RS, Glanville JM, Higgins JP, Kirychuk SP, Sargeant JM, Totton SC, Wood H, Von Essen SG. Updated systematic review: associations between proximity to animal feeding operations and health of individuals in nearby communities. Syst Rev. 2017;6:86.
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355:I4919.
Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122:711.
Johnson PI, Koustas E, Vesterinen HM, Sutton P, Atchley DS, Kim AN, Campbell M, Donald JM, Sen S, Bero L. Application of the Navigation Guide systematic review methodology to the evidence for developmental and reproductive toxicity of triclosan. Environ Int. 2016;92:716–28.
Koustas E, Lam J, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: systematic review of nonhuman evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1015.
Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1040.
Lam J, Sutton P, Kalkbrenner A, Windham G, Halladay A, Koustas E, Lawler C, Davidson L, Daniels N, Newschaffer C. A systematic review and meta-analysis of multiple airborne pollutants and autism spectrum disorder. PLoS One. 2016;11:E0161851.
Vesterinen HM, Johnson PI, Atchley DS, Sutton P, Lam J, Zlatnik MG, Sen S, Woodruff TJ. Fetal growth and maternal glomerular filtration rate: a systematic review. J Matern Fetal Neonatal Med. 2015;28:2176–81.
Carrel M, Schweizer ML, Sarrazin MV, Smith TC, Perencevich EN. Residential proximity to large numbers of swine in feeding operations is associated with increased risk of methicillin-resistant Staphylococcus aureus colonization at time of hospital admission in rural Iowa veterans. Infect Control Hosp Epidemiol. 2014;35:190–2.
Casey JA, Curriero FC, Cosgrove SE, Nachman KE, Schwartz BS. High-density livestock operations, crop field application of manure, and risk of community-associated methicillin-resistant Staphylococcus aureus infection in Pennsylvania. JAMA Intern Med. 2013;173:1980–90.
Davis MA, Moore DL, Baker KN, French NP, Patnode M, Hensley J, Macdonald K, Besser TE. risk factors for campylobacteriosis in two Washington state counties with high numbers of dairy farms. J Clin Microbiol. 2013;51:3921–7.
Pavilonis B, Sanderson W, Merchant JA. Relative exposure to swine animal feeding operations and childhood asthma prevalence in an agricultural cohort. Environ Res. 2013;122:74–80.
Wing S, Horton RA, Marshall SW, Thu K, Tajik M, Schinasi L, Schiffman SS. Air pollution and odor in communities near industrial swine operations. Environ Health Perspect. 2008;116:1362.
Casey JA, Shopsin B, Cosgrove SE, Nachman KE, Curriero FC, Rose HR, Schwartz BS. High-density livestock production and molecularly characterized MRSA infections in Pennsylvania. Environ Health Perspect. 2014;122:464.
Hansen SF. Late lessons from early warnings: science, precaution, innovation (Eea Report No 1/2013) 2013 [Available At: Https://www.Eea.Europa.Eu/Publications/Late-Lessons-2. Accessed: 31 May 2017].
Price LB, Graham JP, Lackey LG, Roess A, Vailes R, Silbergeld E. Elevated risk of carrying gentamicin-resistant Escherichia coli among US poultry workers. Environ Health Perspect. 2007:1738–42.
Smith TC, Male MJ, Harper AL, Kroeger JS, Tinkler GP, Moritz ED, Capuano AW, Herwaldt LA, Diekema DJ. Methicillin-resistant Staphylococcus aureus (MRSA) strain ST398 is present in midwestern US swine and swine workers. PLoS One. 2009;4:E4258.
Hatcher SM, Rhodes SM, Stewart JR, Silbergeld E, Pisanic N, Larsen J, Jiang S, Krosche A, Hall D, Carroll KC. The prevalence of antibiotic-resistant Staphylococcus aureus nasal carriage among industrial hog operation workers, community residents, and children living in their households: North Carolina, USA. Environ Health Perspect. 2017;125:560.
Rinsky JL, Nadimpalli M, Wing S, Hall D, Baron D, Lb P, Larsen J, Stegger M, Stewart J, Heaney CD. Livestock-associated methicillin and multidrug resistant Staphylococcus aureus is present among industrial, not antibiotic-free livestock operation workers in North Carolina. PLoS One. 2013;8:E67641.
Gray GC, Mccarthy T, Capuano AW, Setterquist SF, Olsen CW, Alavanja MC, Lynch CF. Swine workers and swine influenza virus infections. Emerg Infect Dis. 2007;13:1871.
Lantos PM, Hoffman K, Höhle M, Anderson B, Gray GC. Are people living near modern swine production facilities at increased risk of influenza virus infection? Clin Infect Dis. 2016;63:1558–63.
Heederik D, Sigsgaard T, Thorne PS, Kline JN, Avery R, Bønløkke JH, Chrischilles EA, Dosman JA, Duchaine C, Kirkhorn SR. Health effects of airborne exposures from concentrated animal feeding operations. Environ Health Perspect. 2007:298–302.
Feingold BJ, Silbergeld EK, Curriero FC, Van Cleef BA, Heck ME, Kluytmans JA. Livestock density as risk factor for livestock-associated methicillin-resistant Staphylococcus Aureus, the Netherlands. Emerg Infect Dis. 2012;18:1841.
Avery RC, Wing S, Marshall SW, Schiffman SS. Odor from industrial hog farming operations and mucosal immune function in neighbors. Arch Environ Health. 2004;59:101–8.
Schinasi L, Horton RA, Guidry VT, Wing S, Marshall SW, Morland KB. Air pollution, lung function, and physical symptoms in communities near concentrated swine feeding operations. Epidemiology. 2011;22:208–15.
Arah OA, Chiba Y, Greenland S. Bias formulas for external adjustment and sensitivity analysis of unmeasured confounders. Ann Epidemiol. 2008;18:637–46.
Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB, Wynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. Int J Epidemiol. 2009;38:1175–91.
Horton RA, Wing S, Marshall SW, Brownley KA. Malodor as a trigger of stress and negative mood in neighbors of industrial hog operations. Am J Public Health. 2009;99:S610–5.
Wing S, Horton RA, Rose KM. Air pollution from industrial swine operations and blood pressure of neighboring residents. Environ Health Perspect. 2013;121:92.
Pd A. Fixed effects regression models: Sage Publications; 2009.
Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. 1991;133:144–53.
Radon K, Schulze A, Ehrenstein V, Van Strien RT, Praml G, Nowak D. Environmental exposure to confined animal feeding operations and respiratory health of neighboring residents. Epidemiology. 2007;18:300–8.
Schulze A, Römmelt H, Ehrenstein V, Van Strien R, Praml G, Küchenhoff H, Nowak D, Radon K. Effects on pulmonary health of neighboring residents of concentrated animal feeding operations: exposure assessed using optimized estimation technique. Arch Environ Occup Health. 2011;66:146–54.
Kim BF, Laestadius LI, Lawrence RS, Martin RP, Mckenzie SE, Nachman KE, Smith TJS, Truant P. Industrial food animal production in America: examining the impact of the Pew Commission’s priority recommendations 2013 [Available At: Http://www.Jhsph.Edu/Research/Centers-And-Institutes/Johns-Hopkins-Center-For-A-Livable-Future/_Pdf/Research/Clf_Reports/Clf-Pew-For%20web.Pdf. Accessed: 1 June 2017].
National Resources Defense Council. Antibiotic resistance: from the farm to you 2015 [Available At: Https://www.Nrdc.Org/Sites/Default/Files/Antibiotic-Resistance-Farms-Fs.Pdf. Accessed: 1 June 2017].
Maryland Campaign To Keep Antibiotics Working. Reports: a selected bibliography of resources 2017 [Available At: Https://Mdkeepabxworking.Wordpress.Com/Reports-A-Selected-Bibliography-Of-Resources/. Accessed: 1 June 2017].
California State Legislature. SB-27 livestock: use of antimicrobial drugs 2015 [Available At: Https://Leginfo.Legislature.Ca.Gov/Faces/Billnavclient.Xhtml?Bill_Id=201520160sb27. Accessed: 1 June 2017].
Maryland General Assembly. SB0422: Keep Antibiotics Effective Act Of 2017 2017 [Available At: Http://Mgaleg.Maryland.Gov/Webmga/Frmmain.Aspx?Pid=Billpage&Tab=Subject3&Id=Sb0422&Stab=01&Ys=2017rs. Accessed: 1 June 2017].
Casey JA, Kim BF, Larsen J, Price LB, Nachman KE. Industrial food animal production and community health. Curr Environ Health Rep. 2015;2:259–71.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Nachman, K.E., Lam, J., Schinasi, L.H. et al. O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions. Syst Rev 6, 179 (2017). https://doi.org/10.1186/s13643-017-0575-7