Authors’ response to comments from Nachman KE et al.
Systematic Reviews volume 6, Article number: 210 (2017)
Authors’ response to comments letter to the editor from Nachman KE et al.
That our systematic review has generated debate is not surprising, as this is an important and divisive topic . Nachman KE et al.  stated that they would have conducted the review with different eligibility criteria and risk-of-bias assessment. It is to be expected that different review teams will make slightly different judgements about the breadth of their review and the types of studies they consider appropriate to include. Our criteria were specified in advance and described in our review protocol . We would welcome other groups undertaking a similar review with different eligibility criteria to investigate whether the findings are substantially different. It is important that such reviews take a systematic approach and are reported transparently, like ours, to overcome problems of narrative syntheses that have been carried out in this area in the past. We invite readers to read the Open Peer Reports of our original submission for more details of the decisions we made, including discussion about how we considered residual confounding in observational studies, the use of prevalent outcomes, chronic and acute disease, multiplicity, the age of data reported in studies, and effect measures obtained from observational studies. Below, we address specific comments made by Nachman KE et al. .
The authors critique our choice of risk-of-bias (ROB) tool. We selected the tool that we considered to be the most rigorous tool available for non-randomized studies (including observational designs) at the time of writing our protocol, the ROBINS-I tool . The tool has been carefully developed by a large team and is based on the latest thinking in causal inference. Contrary to the impression portrayed by Nachman KE et al. , we modified the tool to ensure applicability to non-interventional exposures. For example, we omitted the domain that asks about adherence to interventions, we modified some elements, and we added details about exposure measurement. Nachman KE et al.  question our inclusion of a question about whether post-intervention variables could influence study participation. To rule out selection bias, however, it is important to determine that post-exposure variables did not influence study participation. We relayed our experiences with the tool to the authors of ROBINS-I so that our feedback could be incorporated into ongoing development of a tool to assess risk of bias in exposure studies . We note that the draft version of this new tool for exposure studies is remarkably similar to ROBINS-I, so we do not agree with Nachman KE et al.  that a different approach is required in environmental health research. Nachman KE et al.  appear to place a higher evidentiary value on observational studies than we are willing to place. We do not believe that unavoidable limitations in observational studies lead to lower risks of bias in their findings.
Nachman KE et al.  also propose different eligibility criteria and questioned the exclusion of studies with only one unit of exposure. We regret that our rationale for exclusion was not clearer, so we will provide more explanation here. The concept is perhaps best illustrated using experimental design concepts. Consider a researcher interested in the comparative effects of fertilizer A and B on seed length who allocates one plant to receive A and B, and another plant to receive neither. If the researcher then measured the length of each seed produced by each plant, this would represent replication since there are multiple seeds from each plant. However, the multiple seeds would not allow stronger conclusions to be made about the effects of the fertilizers since the researcher would not be able to attribute any differences between the plants observed in the seed length to fertilizer A, fertilizer B, or other unmeasured differences such as amount of sunlight received by the plants, or random chance. Clearly, the researchers should have included multiple plants for each exposure and adjusted for the correlated outcomes from seeds within the plants. The same principle applies here, although it might be harder to detect at the macro level. For any observational study, when a single unit of exposure is used (for this topic, a single farm location compared to a single non-farm location), although the health outcome is measured on many individuals (i.e., the equivalent to the seeds), there is no replication of the exposure variable and therefore no variation in the exposure, and subsequently, no meaningful statistical analysis or causal inference. Consequently, as with the plant experiment, all of the characteristics of that unit of exposure (i.e., the farm location) will have the same association with the outcome. Further, by means of a simple but extreme example, if a study that uses a single farm location as the exposure substituted completely unrelated factors such as barn color (or owner’s surname) as the exposure variable, the data analysis would find the same association, although clearly causal inference is mistaken. To avoid this bias, we excluded studies with this characteristic.
With respect to the exclusion of the manuscript by Casey JA et al. , based on the approach to reporting (see Table 1), we were unable to determine whether or not people who had occupational exposure (i.e., farmers) were excluded from the study population, and therefore, we excluded the study. Other studies were more explicit that only community members were studied. As some authors of this comment are also authors of Casey JA et al.  (K. E. Nachman and Joan A. Casey), it is likely they have better knowledge of the study population than we were able to infer from the details reported in the publication.
Other minor issues relate to specific studies; for example, the inclusion of Avery RC et al.  as a study from the Community Health Effects of Industrial Hog Operations (CHEIHO) study, rather than as part of a smaller pilot investigation, is unlikely to impact our inference, but we thank the authors for pointing this out to us and to readers of Systematic Reviews. The authors point out that we should have been more precise when discussing the cross-sectional nature of the study by saying “many” of the studies were cross-sectional. On review of the extracted data, the longitudinal studies that were cited were designated as cohort studies and the risk of bias assessed as such [7,8,9].
This topic and these data are an illustration of why the scientific community should embrace the systematic review methodology and debate about the interpretation of study results. We encourage the discussion about how to assess the risk of bias in observational studies and the evidentiary value of such studies. Given the recent modification of the risk-of-bias tool for interventions for randomized studies (which are far easier to evaluate), the current tool for environmental health with the proposed name ROBINS-E may not be the final word on how to assess the risk of bias in observational studies, and open debate and discussion should be encouraged [5, 10,11,12].
O'Connor AM, Auvermann BW, Dzikamunhenga RS, Glanville JM, Higgins JPT, Kirychuk SP, Sargeant JM, Totton SC, Wood H, Von Essen SG. Updated systematic review: associations between proximity to animal feeding operations and health of individuals in nearby communities. Syst Rev. 2017;6:86.
Nachman KE, Lam JL, Schinasi L, Smith T, Feingold BJ, Casey JA. O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions. Syst Rev. 2017;6:179.
O'Connor AM, Auvermann BW, Higgins JP, Kirychuk SP, Sargeant JM, Von Essen SG, Glanville JM, Wood H. The association between proximity to animal-feeding operations and community health: a protocol for updating a systematic review. Syst Rev. 2014;3:99.
Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
Morgan RL, Sterne JA, Thayer KA. A new instrument to assess risk of bias in non-randomised studies of exposures (ROBINS-E): application to studies of environmental exposure. In: Global evidence summary. Capetown; 2017.
Casey JA, Curriero FC, Cosgrove SE, Nachman KE, Schwartz BS. High-density livestock operations, crop field application of manure, and risk of community-associated methicillin-resistant Staphylococcus aureus infection in Pennsylvania. JAMA Intern Med. 2013;173:1980–90. hhttps://www.globalevidencesummit.org/workshops/new-instrument-assess-risk-bias-non-randomised-studies-exposures-robins-e-application.
Avery RC, Wing S, Marshall SW, Schiffman SS. Odor from industrial hog farming operations and mucosal immune function in neighbors. Arch Environ Health. 2004;59:101–8.
Schinasi L, Horton RA, Guidry VT, Wing S, Marshall SW, Morland KB. Air pollution, lung function, and physical symptoms in communities near concentrated swine feeding operations. Epidemiology. 2011;22:208–15.
Wing S, Horton RA, Rose KM. Air pollution from industrial swine operations and blood pressure of neighboring residents. Environ Health Perspect. 2013;121:92–6.
Higgins J, Sterne JA, Savović J, Page M, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials. In: Chandler J, McKenzie J, Boutron I, Welch V, editors. Cochrane methods Cochrane database of systematic reviews 2016; 2016. Issue 10 (Suppl 1) dxdoiorg/101002/14651858CD201601.
Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, Guyatt G, Hooijmans C, Langendam M, Mandrioli D, et al. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92-93:611–6.
Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, Ratcliffe JM, Kraft AD, Schunemann HJ, Schwingl P, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92-93:617–29.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
JPTH is a co-author of the ROBINS-I tool.
AOC, BWA, RSD, JMG, SPK, JMS, HW, and SCVE declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
O’Connor, A.M., Auvermann, B.W., Dzikamunhenga, R.S. et al. Authors’ response to comments from Nachman KE et al.. Syst Rev 6, 210 (2017). https://doi.org/10.1186/s13643-017-0611-7