Skip to main content

Authors’ response to comments from Nachman KE et al.


Authors’ response to comments letter to the editor from Nachman KE et al.

Peer Review reports

That our systematic review has generated debate is not surprising, as this is an important and divisive topic [1]. Nachman KE et al. [2] stated that they would have conducted the review with different eligibility criteria and risk-of-bias assessment. It is to be expected that different review teams will make slightly different judgements about the breadth of their review and the types of studies they consider appropriate to include. Our criteria were specified in advance and described in our review protocol [3]. We would welcome other groups undertaking a similar review with different eligibility criteria to investigate whether the findings are substantially different. It is important that such reviews take a systematic approach and are reported transparently, like ours, to overcome problems of narrative syntheses that have been carried out in this area in the past. We invite readers to read the Open Peer Reports of our original submission for more details of the decisions we made, including discussion about how we considered residual confounding in observational studies, the use of prevalent outcomes, chronic and acute disease, multiplicity, the age of data reported in studies, and effect measures obtained from observational studies. Below, we address specific comments made by Nachman KE et al. [2].

The authors critique our choice of risk-of-bias (ROB) tool. We selected the tool that we considered to be the most rigorous tool available for non-randomized studies (including observational designs) at the time of writing our protocol, the ROBINS-I tool [4]. The tool has been carefully developed by a large team and is based on the latest thinking in causal inference. Contrary to the impression portrayed by Nachman KE et al. [2], we modified the tool to ensure applicability to non-interventional exposures. For example, we omitted the domain that asks about adherence to interventions, we modified some elements, and we added details about exposure measurement. Nachman KE et al. [2] question our inclusion of a question about whether post-intervention variables could influence study participation. To rule out selection bias, however, it is important to determine that post-exposure variables did not influence study participation. We relayed our experiences with the tool to the authors of ROBINS-I so that our feedback could be incorporated into ongoing development of a tool to assess risk of bias in exposure studies [5]. We note that the draft version of this new tool for exposure studies is remarkably similar to ROBINS-I, so we do not agree with Nachman KE et al. [2] that a different approach is required in environmental health research. Nachman KE et al. [2] appear to place a higher evidentiary value on observational studies than we are willing to place. We do not believe that unavoidable limitations in observational studies lead to lower risks of bias in their findings.

Nachman KE et al. [2] also propose different eligibility criteria and questioned the exclusion of studies with only one unit of exposure. We regret that our rationale for exclusion was not clearer, so we will provide more explanation here. The concept is perhaps best illustrated using experimental design concepts. Consider a researcher interested in the comparative effects of fertilizer A and B on seed length who allocates one plant to receive A and B, and another plant to receive neither. If the researcher then measured the length of each seed produced by each plant, this would represent replication since there are multiple seeds from each plant. However, the multiple seeds would not allow stronger conclusions to be made about the effects of the fertilizers since the researcher would not be able to attribute any differences between the plants observed in the seed length to fertilizer A, fertilizer B, or other unmeasured differences such as amount of sunlight received by the plants, or random chance. Clearly, the researchers should have included multiple plants for each exposure and adjusted for the correlated outcomes from seeds within the plants. The same principle applies here, although it might be harder to detect at the macro level. For any observational study, when a single unit of exposure is used (for this topic, a single farm location compared to a single non-farm location), although the health outcome is measured on many individuals (i.e., the equivalent to the seeds), there is no replication of the exposure variable and therefore no variation in the exposure, and subsequently, no meaningful statistical analysis or causal inference. Consequently, as with the plant experiment, all of the characteristics of that unit of exposure (i.e., the farm location) will have the same association with the outcome. Further, by means of a simple but extreme example, if a study that uses a single farm location as the exposure substituted completely unrelated factors such as barn color (or owner’s surname) as the exposure variable, the data analysis would find the same association, although clearly causal inference is mistaken. To avoid this bias, we excluded studies with this characteristic.

With respect to the exclusion of the manuscript by Casey JA et al. [6], based on the approach to reporting (see Table 1), we were unable to determine whether or not people who had occupational exposure (i.e., farmers) were excluded from the study population, and therefore, we excluded the study. Other studies were more explicit that only community members were studied. As some authors of this comment are also authors of Casey JA et al. [6] (K. E. Nachman and Joan A. Casey), it is likely they have better knowledge of the study population than we were able to infer from the details reported in the publication.

Table 1 Text reproduced from Casey JA et al. [6]

Other minor issues relate to specific studies; for example, the inclusion of Avery RC et al. [7] as a study from the Community Health Effects of Industrial Hog Operations (CHEIHO) study, rather than as part of a smaller pilot investigation, is unlikely to impact our inference, but we thank the authors for pointing this out to us and to readers of Systematic Reviews. The authors point out that we should have been more precise when discussing the cross-sectional nature of the study by saying “many” of the studies were cross-sectional. On review of the extracted data, the longitudinal studies that were cited were designated as cohort studies and the risk of bias assessed as such [7,8,9].

This topic and these data are an illustration of why the scientific community should embrace the systematic review methodology and debate about the interpretation of study results. We encourage the discussion about how to assess the risk of bias in observational studies and the evidentiary value of such studies. Given the recent modification of the risk-of-bias tool for interventions for randomized studies (which are far easier to evaluate), the current tool for environmental health with the proposed name ROBINS-E may not be the final word on how to assess the risk of bias in observational studies, and open debate and discussion should be encouraged [5, 10,11,12].


  1. O'Connor AM, Auvermann BW, Dzikamunhenga RS, Glanville JM, Higgins JPT, Kirychuk SP, Sargeant JM, Totton SC, Wood H, Von Essen SG. Updated systematic review: associations between proximity to animal feeding operations and health of individuals in nearby communities. Syst Rev. 2017;6:86.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Nachman KE, Lam JL, Schinasi L, Smith T, Feingold BJ, Casey JA. O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions. Syst Rev. 2017;6:179.

    Article  PubMed  PubMed Central  Google Scholar 

  3. O'Connor AM, Auvermann BW, Higgins JP, Kirychuk SP, Sargeant JM, Von Essen SG, Glanville JM, Wood H. The association between proximity to animal-feeding operations and community health: a protocol for updating a systematic review. Syst Rev. 2014;3:99.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Morgan RL, Sterne JA, Thayer KA. A new instrument to assess risk of bias in non-randomised studies of exposures (ROBINS-E): application to studies of environmental exposure. In: Global evidence summary. Capetown; 2017.

  6. Casey JA, Curriero FC, Cosgrove SE, Nachman KE, Schwartz BS. High-density livestock operations, crop field application of manure, and risk of community-associated methicillin-resistant Staphylococcus aureus infection in Pennsylvania. JAMA Intern Med. 2013;173:1980–90. h

  7. Avery RC, Wing S, Marshall SW, Schiffman SS. Odor from industrial hog farming operations and mucosal immune function in neighbors. Arch Environ Health. 2004;59:101–8.

    Article  CAS  PubMed  Google Scholar 

  8. Schinasi L, Horton RA, Guidry VT, Wing S, Marshall SW, Morland KB. Air pollution, lung function, and physical symptoms in communities near concentrated swine feeding operations. Epidemiology. 2011;22:208–15.

    Article  PubMed  Google Scholar 

  9. Wing S, Horton RA, Rose KM. Air pollution from industrial swine operations and blood pressure of neighboring residents. Environ Health Perspect. 2013;121:92–6.

    PubMed  Google Scholar 

  10. Higgins J, Sterne JA, Savović J, Page M, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials. In: Chandler J, McKenzie J, Boutron I, Welch V, editors. Cochrane methods Cochrane database of systematic reviews 2016; 2016. Issue 10 (Suppl 1) dxdoiorg/101002/14651858CD201601.

    Google Scholar 

  11. Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, Guyatt G, Hooijmans C, Langendam M, Mandrioli D, et al. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92-93:611–6.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, Ratcliffe JM, Kraft AD, Schunemann HJ, Schwingl P, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92-93:617–29.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


Not applicable


Not applicable

Availability of data and materials

Not applicable

Author information

Authors and Affiliations



AOC drafted and distributed the response. BWA, RSD, JPTH, JMG, SPK, JMS, ST, HW, SCVE contributed sections, revisions and insights. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Annette M. O’Connor.

Ethics declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

JPTH is a co-author of the ROBINS-I tool.

AOC, BWA, RSD, JMG, SPK, JMS, HW, and SCVE declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

O’Connor, A.M., Auvermann, B.W., Dzikamunhenga, R.S. et al. Authors’ response to comments from Nachman KE et al.. Syst Rev 6, 210 (2017).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: