Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions

  • Keeve E. Nachman1, 2, 3, 4Email author,
  • Juleen Lam5,
  • Leah H. Schinasi6, 7,
  • Tara C. Smith8, 9, 10,
  • Beth J. Feingold11 and
  • Joan A. Casey12
Systematic Reviews20176:179

https://doi.org/10.1186/s13643-017-0575-7

Received: 28 June 2017

Accepted: 22 August 2017

Published: 31 August 2017

Abstract

In this comment, we summarize several scientific concerns with the recently published systematic review from O’Connor and colleagues that examined the relationship between proximity to animal-feeding operations and health of individuals in nearby communities. The authors utilized a bias tool not designed for environmental health research, erroneously excluded important studies, and incorrectly interpreted others. As a result, the conclusions drawn in the review misrepresent the evidence from the published literature, limiting its value to policymakers, researchers, and the public.

There is increasing recognition of the value of systematic reviews in the context of decision-making. These reviews are a popular tool in evidence-based medicine [1], and potential opportunities for their application in judging environmental health risks are actively discussed [25]. When properly conducted, systematic reviews can serve as critical inputs to the policy process and result in choices about interventions that are in the public’s interest and based on the best available science. Conversely, the use of systematic reviews that suffer from critical design flaws or lack rigorous evaluation of the evidence may hurt public health. We have concern that the recently published systematic review by O’Connor and colleagues [6] on the associations between living near an animal-feeding operation (AFO) and human health falls into the latter category. Although the review integrates some best practice guidelines for conducting systematic reviews, such as developing and registering a protocol beforehand in PROSPERO, the review appears substantially flawed in its approach and conduct; the authors utilized a bias tool that is inappropriate for environmental health research, erroneously excluded important studies, and incorrectly interpreted others. As a result, the ultimate conclusions of the review misrepresent evidence from the published literature. Given that the authors describe their review as having utility for decision-making and prioritizing research, a formal critique of the work is warranted to help calibrate its value for policymakers, researchers, and the public.

The systematic review suffers from shortcomings related to the application of the ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) tool. The tool was originally developed by the Cochrane Collaboration to evaluate the internal validity of studies of potentially beneficial medical interventions [7]. Direct application to environmental health research is inappropriate because the evidence base and decision-making context differs substantially from clinical settings. For instance, many of the signaling questions to evaluate the study’s internal validity that inquire about intervention-specific details (adherence to the assigned intervention regimen, post-intervention variables influencing study participation, etc.) are not directly relevant to studies of potentially harmful environmental exposures. The use of the ROBINS-I tool in this context will inherently bias conclusions by finding that all included studies are of low quality, given the unavoidable limitations even in well-designed observational epidemiology studies.

Since 2009, there have been ongoing efforts to translate systematic review methods from the clinical sciences to the environmental health decision context, including the development of risk of bias tools in academic and government settings. There are now at least two published methods [2, 8] and several case studies [3, 5, 913] that illustrate approaches more directly applicable to review questions in the environmental health decision-making context than ROBINS-I.

The article-screening process and the application of the exclusion criteria were flawed in design and conduct. In the Level 2 screening, the authors specified that articles that did not include “more than one unit of measurement of exposure” would not be considered further in the review. There is no clear definition of this exclusion criterion offered in the text or justification for why the authors chose to exclude such studies. This criterion was used to eliminate eight studies identified in the Level 1 screening from further consideration. Inclusion of these articles would have increased the number of included studies by 50%, indicating that the impact of such an unjustified exclusion may be so strong in this case as to have led to spurious conclusions.

Beyond design flaws, we noted that numerous key papers examining infectious disease and respiratory outcomes related to community exposures to AFOs [1418] were identified during the Level 1 screening but inappropriately excluded during the Level 2 screening. The rationale presented for excluding these papers was reported in the supplemental material as “the unit of analysis was not at the individual human level or the study looked at occupational exposure only”; however, neither of these statements applied to any of these studies. For example, Casey et al.’s analyses drew on individual-level longitudinal electronic health record data from primary care patients of the Geisinger Health System in Pennsylvania [15]. This study was the first in the USA to demonstrate an association between residential proximity to swine AFOs and manure-applied crop fields and methicillin-resistant Staphylococcus aureus (MRSA) infection in a general population sample. The inappropriate exclusion of studies highlights serious concerns related to the rigor and consistency of analysis at the second level of screening and calls into question the validity of conclusions drawn by the authors on the basis of their review. In addition, we note that at least one of our papers [19] that appears relevant to the primary research question was not identified during the Level 1 screening process. Problems with the search strategy may suggest that other key publications could have also been missed, further limiting confidence in the authors’ conclusions.

The authors eliminated ecological studies from their review a priori. A better option may have been to include them while acknowledging the weaknesses and strengths; while inferences about individual-level effects of exposures cannot be inferred from ecologic analyses, these studies have utility in that they establish preliminary evidence of potential associations between exposure and health outcomes, often over large geographies. The authors also limited the scope of their review to the health effects in the community rather than including occupational exposures, despite the notion that workers often reside in communities with their families. While we do not take issue with the exclusion of occupational studies from the systematic review, we note that the environmental health literature offers many examples (e.g., lead, chromium, beryllium) of instances where early warnings arose from higher-level exposures seen in occupational settings [20]. In the case of AFOs, a growing literature exists documenting relationships between occupational exposures and infectious disease and respiratory outcomes [2127].

Multiple errors in study interpretation and presentation were also noted. For example, the authors stated that Feingold et al. [28] reported an association between MRSA carriage and livestock density, but the Feingold study utilized a case-case analysis comparing MRSA ST-398 to other MRSA strains, not to a control population that would allow such a conclusion to be drawn. Other examples include the erroneous presentation of data from the Avery et al. [29] study, which O’Connor et al. mistakenly list in Table 1 as part of the Community Health Effects of Industrial Hog Operations (CHEIHO) study, rather than as part of a smaller pilot investigation. Also contrary to its presentation in Table 1, Schinasi et al. [30] employed multivariate fixed effects regression models as the primary analytic method.

O’Connor et al.’s evaluation of risk of bias across studies may reflect a misunderstanding of the design and analysis used in some of the papers reported. The authors state, “[b]ecause the studies were cross-sectional in nature, adjustment for confounders would not provide protection against residual confounding.” We disagree with these assertions for two reasons. First, methods exist to evaluate the degree of bias introduced from unmeasured confounders [31] and this residual confounding does not always prevent causal inference from cross-sectional studies [32]. Second, multiple studies included in the review were longitudinal in nature [30, 33, 34]. For example, the CHEIHO study used a case-crossover design and adjusted for within-person, time-invariant confounders. As a result, the models quantified the impact of exposure variability on response variability within each person and, in so doing, adjusted for all stable, potentially confounding covariates that varied between people [35]. While it is true that unmeasured time-varying confounding may remain, these models accounted for much of the confounding that is normally a concern [36].

In their discussion section, O’Connor and colleagues call into question studies that are older than 10 years (e.g., [30, 33, 34, 37, 38]). The authors suggest that “dramatic” changes in environmental regulation, animal housing, and manure management may have occurred that would invalidate these studies. However, no evidence supporting this point is presented. Additionally, there is no reason to believe that biologic effects of exposures related to AFOs would change as a function of time.

Policymakers depend on peer-reviewed scientific research as a credible source of information upon which decisions can be built. As a recent example, some of the studies of community risks of AFOs that were excluded in this review [14, 15] and other related research syntheses [39] played an important role in advocacy efforts (e.g., [40, 41]) around two state policies aimed at eliminating non-therapeutic antibiotic use in animal agriculture [42, 43]. Despite the utility of these studies, keeping up with and interpreting the scientific literature is beyond the capacity of most policymakers; thus, the conclusions of systematic reviews can be instrumental in supporting the development of evidence-based policy. When design flaws and less-than-rigorous conduct influence their results, however, the findings of these reviews may be used to adversely impact the policy process or resultant interventions. In contrast to the findings of the authors of this review, we believe there is a considerable and growing body of rigorously conducted scientific evidence that suggests connections between living near AFOs and adverse health outcomes [44]. With a different tool to evaluate the evidence, and a more rigorous application of that tool, it is entirely possible the authors would have come to a different conclusion regarding the community health risks of AFOs.

Declarations

Authors’ contributions

All authors discussed and decided upon the main themes of the commentary. KN led the initial outlining of the commentary; JL, LS, TS, BF and JC contributed sections, revisions and insights. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Johns Hopkins Center for a Livable Future, Johns Hopkins University
(2)
Department of Environmental Health and Engineering, Bloomberg School of Public Health, Johns Hopkins University
(3)
Johns Hopkins Risk Sciences and Public Policy Institute
(4)
Department of Health Policy and Management, Bloomberg School of Public Health, Johns Hopkins University
(5)
Program on Reproductive Health and the Environment, Department of OB/GYN & RS, University of California
(6)
Department of Environmental and Occupational Health, Dornsife School of Public Health, Drexel University
(7)
Urban Health Collaborative, Dornsife School of Public Health, Drexel University
(8)
Department of Epidemiology, University of Iowa College of Public Health
(9)
Center for Emerging Infectious Diseases, University of Iowa College of Public Health
(10)
Department of Biostatistics, Environmental Health Sciences and Epidemiology, College of Public Health, Kent State University
(11)
Department of Environmental Health Sciences, University at Albany School of Public Health, State University of New York
(12)
School of Public Health, University of California

References

  1. Ioannidis J. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514.View ArticlePubMedGoogle Scholar
  2. Woodruff TJ, Sutton P. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect. 2014;122:1007.PubMedPubMed CentralGoogle Scholar
  3. Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122Google Scholar
  4. Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, Guyatt G, Hooijmans C, Langendam M, Mandrioli D. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92:611–6.View ArticlePubMedGoogle Scholar
  5. Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, Ratcliffe JM, Kraft AD, Schünemann HJ, Schwingl P, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92–93:617–29.View ArticlePubMedPubMed CentralGoogle Scholar
  6. O’connor AM, Auvermann BW, Dzikamunhenga RS, Glanville JM, Higgins JP, Kirychuk SP, Sargeant JM, Totton SC, Wood H, Von Essen SG. Updated systematic review: associations between proximity to animal feeding operations and health of individuals in nearby communities. Syst Rev. 2017;6:86.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355:I4919.View ArticleGoogle Scholar
  8. Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122:711.PubMedPubMed CentralGoogle Scholar
  9. Johnson PI, Koustas E, Vesterinen HM, Sutton P, Atchley DS, Kim AN, Campbell M, Donald JM, Sen S, Bero L. Application of the Navigation Guide systematic review methodology to the evidence for developmental and reproductive toxicity of triclosan. Environ Int. 2016;92:716–28.View ArticlePubMedGoogle Scholar
  10. Koustas E, Lam J, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: systematic review of nonhuman evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1015.PubMedPubMed CentralGoogle Scholar
  11. Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA, Axelrad DA, Woodruff TJ. The Navigation Guide—evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1040.PubMedPubMed CentralGoogle Scholar
  12. Lam J, Sutton P, Kalkbrenner A, Windham G, Halladay A, Koustas E, Lawler C, Davidson L, Daniels N, Newschaffer C. A systematic review and meta-analysis of multiple airborne pollutants and autism spectrum disorder. PLoS One. 2016;11:E0161851.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Vesterinen HM, Johnson PI, Atchley DS, Sutton P, Lam J, Zlatnik MG, Sen S, Woodruff TJ. Fetal growth and maternal glomerular filtration rate: a systematic review. J Matern Fetal Neonatal Med. 2015;28:2176–81.View ArticlePubMedGoogle Scholar
  14. Carrel M, Schweizer ML, Sarrazin MV, Smith TC, Perencevich EN. Residential proximity to large numbers of swine in feeding operations is associated with increased risk of methicillin-resistant Staphylococcus aureus colonization at time of hospital admission in rural Iowa veterans. Infect Control Hosp Epidemiol. 2014;35:190–2.View ArticlePubMedGoogle Scholar
  15. Casey JA, Curriero FC, Cosgrove SE, Nachman KE, Schwartz BS. High-density livestock operations, crop field application of manure, and risk of community-associated methicillin-resistant Staphylococcus aureus infection in Pennsylvania. JAMA Intern Med. 2013;173:1980–90.View ArticlePubMedPubMed CentralGoogle Scholar
  16. Davis MA, Moore DL, Baker KN, French NP, Patnode M, Hensley J, Macdonald K, Besser TE. risk factors for campylobacteriosis in two Washington state counties with high numbers of dairy farms. J Clin Microbiol. 2013;51:3921–7.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Pavilonis B, Sanderson W, Merchant JA. Relative exposure to swine animal feeding operations and childhood asthma prevalence in an agricultural cohort. Environ Res. 2013;122:74–80.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Wing S, Horton RA, Marshall SW, Thu K, Tajik M, Schinasi L, Schiffman SS. Air pollution and odor in communities near industrial swine operations. Environ Health Perspect. 2008;116:1362.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Casey JA, Shopsin B, Cosgrove SE, Nachman KE, Curriero FC, Rose HR, Schwartz BS. High-density livestock production and molecularly characterized MRSA infections in Pennsylvania. Environ Health Perspect. 2014;122:464.PubMedPubMed CentralGoogle Scholar
  20. Hansen SF. Late lessons from early warnings: science, precaution, innovation (Eea Report No 1/2013) 2013 [Available At: Https://www.Eea.Europa.Eu/Publications/Late-Lessons-2. Accessed: 31 May 2017].
  21. Price LB, Graham JP, Lackey LG, Roess A, Vailes R, Silbergeld E. Elevated risk of carrying gentamicin-resistant Escherichia coli among US poultry workers. Environ Health Perspect. 2007:1738–42.Google Scholar
  22. Smith TC, Male MJ, Harper AL, Kroeger JS, Tinkler GP, Moritz ED, Capuano AW, Herwaldt LA, Diekema DJ. Methicillin-resistant Staphylococcus aureus (MRSA) strain ST398 is present in midwestern US swine and swine workers. PLoS One. 2009;4:E4258.View ArticlePubMedPubMed CentralGoogle Scholar
  23. Hatcher SM, Rhodes SM, Stewart JR, Silbergeld E, Pisanic N, Larsen J, Jiang S, Krosche A, Hall D, Carroll KC. The prevalence of antibiotic-resistant Staphylococcus aureus nasal carriage among industrial hog operation workers, community residents, and children living in their households: North Carolina, USA. Environ Health Perspect. 2017;125:560.PubMedGoogle Scholar
  24. Rinsky JL, Nadimpalli M, Wing S, Hall D, Baron D, Lb P, Larsen J, Stegger M, Stewart J, Heaney CD. Livestock-associated methicillin and multidrug resistant Staphylococcus aureus is present among industrial, not antibiotic-free livestock operation workers in North Carolina. PLoS One. 2013;8:E67641.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Gray GC, Mccarthy T, Capuano AW, Setterquist SF, Olsen CW, Alavanja MC, Lynch CF. Swine workers and swine influenza virus infections. Emerg Infect Dis. 2007;13:1871.View ArticlePubMedPubMed CentralGoogle Scholar
  26. Lantos PM, Hoffman K, Höhle M, Anderson B, Gray GC. Are people living near modern swine production facilities at increased risk of influenza virus infection? Clin Infect Dis. 2016;63:1558–63.View ArticlePubMedGoogle Scholar
  27. Heederik D, Sigsgaard T, Thorne PS, Kline JN, Avery R, Bønløkke JH, Chrischilles EA, Dosman JA, Duchaine C, Kirkhorn SR. Health effects of airborne exposures from concentrated animal feeding operations. Environ Health Perspect. 2007:298–302.Google Scholar
  28. Feingold BJ, Silbergeld EK, Curriero FC, Van Cleef BA, Heck ME, Kluytmans JA. Livestock density as risk factor for livestock-associated methicillin-resistant Staphylococcus Aureus, the Netherlands. Emerg Infect Dis. 2012;18:1841.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Avery RC, Wing S, Marshall SW, Schiffman SS. Odor from industrial hog farming operations and mucosal immune function in neighbors. Arch Environ Health. 2004;59:101–8.View ArticlePubMedGoogle Scholar
  30. Schinasi L, Horton RA, Guidry VT, Wing S, Marshall SW, Morland KB. Air pollution, lung function, and physical symptoms in communities near concentrated swine feeding operations. Epidemiology. 2011;22:208–15.View ArticlePubMedGoogle Scholar
  31. Arah OA, Chiba Y, Greenland S. Bias formulas for external adjustment and sensitivity analysis of unmeasured confounders. Ann Epidemiol. 2008;18:637–46.View ArticlePubMedGoogle Scholar
  32. Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB, Wynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. Int J Epidemiol. 2009;38:1175–91.View ArticlePubMedGoogle Scholar
  33. Horton RA, Wing S, Marshall SW, Brownley KA. Malodor as a trigger of stress and negative mood in neighbors of industrial hog operations. Am J Public Health. 2009;99:S610–5.View ArticlePubMedPubMed CentralGoogle Scholar
  34. Wing S, Horton RA, Rose KM. Air pollution from industrial swine operations and blood pressure of neighboring residents. Environ Health Perspect. 2013;121:92.PubMedGoogle Scholar
  35. Pd A. Fixed effects regression models: Sage Publications; 2009.Google Scholar
  36. Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. 1991;133:144–53.View ArticlePubMedGoogle Scholar
  37. Radon K, Schulze A, Ehrenstein V, Van Strien RT, Praml G, Nowak D. Environmental exposure to confined animal feeding operations and respiratory health of neighboring residents. Epidemiology. 2007;18:300–8.View ArticlePubMedGoogle Scholar
  38. Schulze A, Römmelt H, Ehrenstein V, Van Strien R, Praml G, Küchenhoff H, Nowak D, Radon K. Effects on pulmonary health of neighboring residents of concentrated animal feeding operations: exposure assessed using optimized estimation technique. Arch Environ Occup Health. 2011;66:146–54.View ArticlePubMedGoogle Scholar
  39. Kim BF, Laestadius LI, Lawrence RS, Martin RP, Mckenzie SE, Nachman KE, Smith TJS, Truant P. Industrial food animal production in America: examining the impact of the Pew Commission’s priority recommendations 2013 [Available At: Http://www.Jhsph.Edu/Research/Centers-And-Institutes/Johns-Hopkins-Center-For-A-Livable-Future/_Pdf/Research/Clf_Reports/Clf-Pew-For%20web.Pdf. Accessed: 1 June 2017].
  40. National Resources Defense Council. Antibiotic resistance: from the farm to you 2015 [Available At: Https://www.Nrdc.Org/Sites/Default/Files/Antibiotic-Resistance-Farms-Fs.Pdf. Accessed: 1 June 2017].
  41. Maryland Campaign To Keep Antibiotics Working. Reports: a selected bibliography of resources 2017 [Available At: Https://Mdkeepabxworking.Wordpress.Com/Reports-A-Selected-Bibliography-Of-Resources/. Accessed: 1 June 2017].
  42. California State Legislature. SB-27 livestock: use of antimicrobial drugs 2015 [Available At: Https://Leginfo.Legislature.Ca.Gov/Faces/Billnavclient.Xhtml?Bill_Id=201520160sb27. Accessed: 1 June 2017].
  43. Maryland General Assembly. SB0422: Keep Antibiotics Effective Act Of 2017 2017 [Available At: Http://Mgaleg.Maryland.Gov/Webmga/Frmmain.Aspx?Pid=Billpage&Tab=Subject3&Id=Sb0422&Stab=01&Ys=2017rs. Accessed: 1 June 2017].
  44. Casey JA, Kim BF, Larsen J, Price LB, Nachman KE. Industrial food animal production and community health. Curr Environ Health Rep. 2015;2:259–71.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2017

Advertisement