This article has Open Peer Review reports available.
Authors’ response to comments from Nachman KE et al.
© The Author(s). 2017
Received: 5 September 2017
Accepted: 16 October 2017
Published: 23 October 2017
Authors’ response to comments letter to the editor from Nachman KE et al.
That our systematic review has generated debate is not surprising, as this is an important and divisive topic . Nachman KE et al.  stated that they would have conducted the review with different eligibility criteria and risk-of-bias assessment. It is to be expected that different review teams will make slightly different judgements about the breadth of their review and the types of studies they consider appropriate to include. Our criteria were specified in advance and described in our review protocol . We would welcome other groups undertaking a similar review with different eligibility criteria to investigate whether the findings are substantially different. It is important that such reviews take a systematic approach and are reported transparently, like ours, to overcome problems of narrative syntheses that have been carried out in this area in the past. We invite readers to read the Open Peer Reports of our original submission for more details of the decisions we made, including discussion about how we considered residual confounding in observational studies, the use of prevalent outcomes, chronic and acute disease, multiplicity, the age of data reported in studies, and effect measures obtained from observational studies. Below, we address specific comments made by Nachman KE et al. .
The authors critique our choice of risk-of-bias (ROB) tool. We selected the tool that we considered to be the most rigorous tool available for non-randomized studies (including observational designs) at the time of writing our protocol, the ROBINS-I tool . The tool has been carefully developed by a large team and is based on the latest thinking in causal inference. Contrary to the impression portrayed by Nachman KE et al. , we modified the tool to ensure applicability to non-interventional exposures. For example, we omitted the domain that asks about adherence to interventions, we modified some elements, and we added details about exposure measurement. Nachman KE et al.  question our inclusion of a question about whether post-intervention variables could influence study participation. To rule out selection bias, however, it is important to determine that post-exposure variables did not influence study participation. We relayed our experiences with the tool to the authors of ROBINS-I so that our feedback could be incorporated into ongoing development of a tool to assess risk of bias in exposure studies . We note that the draft version of this new tool for exposure studies is remarkably similar to ROBINS-I, so we do not agree with Nachman KE et al.  that a different approach is required in environmental health research. Nachman KE et al.  appear to place a higher evidentiary value on observational studies than we are willing to place. We do not believe that unavoidable limitations in observational studies lead to lower risks of bias in their findings.
Nachman KE et al.  also propose different eligibility criteria and questioned the exclusion of studies with only one unit of exposure. We regret that our rationale for exclusion was not clearer, so we will provide more explanation here. The concept is perhaps best illustrated using experimental design concepts. Consider a researcher interested in the comparative effects of fertilizer A and B on seed length who allocates one plant to receive A and B, and another plant to receive neither. If the researcher then measured the length of each seed produced by each plant, this would represent replication since there are multiple seeds from each plant. However, the multiple seeds would not allow stronger conclusions to be made about the effects of the fertilizers since the researcher would not be able to attribute any differences between the plants observed in the seed length to fertilizer A, fertilizer B, or other unmeasured differences such as amount of sunlight received by the plants, or random chance. Clearly, the researchers should have included multiple plants for each exposure and adjusted for the correlated outcomes from seeds within the plants. The same principle applies here, although it might be harder to detect at the macro level. For any observational study, when a single unit of exposure is used (for this topic, a single farm location compared to a single non-farm location), although the health outcome is measured on many individuals (i.e., the equivalent to the seeds), there is no replication of the exposure variable and therefore no variation in the exposure, and subsequently, no meaningful statistical analysis or causal inference. Consequently, as with the plant experiment, all of the characteristics of that unit of exposure (i.e., the farm location) will have the same association with the outcome. Further, by means of a simple but extreme example, if a study that uses a single farm location as the exposure substituted completely unrelated factors such as barn color (or owner’s surname) as the exposure variable, the data analysis would find the same association, although clearly causal inference is mistaken. To avoid this bias, we excluded studies with this characteristic.
Text reproduced from Casey JA et al. 
Case Ascertainment and Control Selection
Incident MRSA cases were identified primarily using laboratory cultures and secondarily by diagnosis codes (eg, International Classification of Diseases, Ninth Revision [ICD-9]) that indicated MRSA infection, as previously described.22 Cases were then classified as either CA-MRSA or HA-MRSA based on presence of health care risk factors (eg, hospitalization, surgery, dialysis, nursing home residence, indwelling device)22,31 or diagnosis more than 2 days after hospital admission using ICD-9 codes21,23,32 and Current Procedural Terminology codes. We then randomly selected patients with SSTI but no history of MRSA using 29 ICD-9 codes (eg, carbuncle, furuncle, abscess)22 and controls with no history of MRSA, and we frequency matched both groups with case patients by age (0-6, 7-18, 19-45, 46-62, 6274, ≥75 years), sex, and diagnosis or an outpatient encounter in the same year as MRSA diagnosis. The SSTI cases were evaluated as a separate case group because some SSTIs occurring during the study period were likely to have been caused by MRSA but not diagnosed as such, and high-density livestock production could cause SSTIs from other bacteria. Therefore, we selected patients with SSTIs without reference to any specific pathogen. If a control had multiple outpatient encounters during the year, a single encounter was randomly selected as the date for exposure assignment.
Other minor issues relate to specific studies; for example, the inclusion of Avery RC et al.  as a study from the Community Health Effects of Industrial Hog Operations (CHEIHO) study, rather than as part of a smaller pilot investigation, is unlikely to impact our inference, but we thank the authors for pointing this out to us and to readers of Systematic Reviews. The authors point out that we should have been more precise when discussing the cross-sectional nature of the study by saying “many” of the studies were cross-sectional. On review of the extracted data, the longitudinal studies that were cited were designated as cohort studies and the risk of bias assessed as such [7–9].
This topic and these data are an illustration of why the scientific community should embrace the systematic review methodology and debate about the interpretation of study results. We encourage the discussion about how to assess the risk of bias in observational studies and the evidentiary value of such studies. Given the recent modification of the risk-of-bias tool for interventions for randomized studies (which are far easier to evaluate), the current tool for environmental health with the proposed name ROBINS-E may not be the final word on how to assess the risk of bias in observational studies, and open debate and discussion should be encouraged [5, 10–12].
Availability of data and materials
AOC drafted and distributed the response. BWA, RSD, JPTH, JMG, SPK, JMS, ST, HW, SCVE contributed sections, revisions and insights. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
JPTH is a co-author of the ROBINS-I tool.
AOC, BWA, RSD, JMG, SPK, JMS, HW, and SCVE declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- O'Connor AM, Auvermann BW, Dzikamunhenga RS, Glanville JM, Higgins JPT, Kirychuk SP, Sargeant JM, Totton SC, Wood H, Von Essen SG. Updated systematic review: associations between proximity to animal feeding operations and health of individuals in nearby communities. Syst Rev. 2017;6:86.View ArticlePubMedPubMed CentralGoogle Scholar
- Nachman KE, Lam JL, Schinasi L, Smith T, Feingold BJ, Casey JA. O’Connor et al. systematic review regarding animal feeding operations and public health: critical flaws may compromise conclusions. Syst Rev. 2017;6:179.View ArticlePubMedPubMed CentralGoogle Scholar
- O'Connor AM, Auvermann BW, Higgins JP, Kirychuk SP, Sargeant JM, Von Essen SG, Glanville JM, Wood H. The association between proximity to animal-feeding operations and community health: a protocol for updating a systematic review. Syst Rev. 2014;3:99.View ArticlePubMedPubMed CentralGoogle Scholar
- Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.View ArticlePubMedPubMed CentralGoogle Scholar
- Morgan RL, Sterne JA, Thayer KA. A new instrument to assess risk of bias in non-randomised studies of exposures (ROBINS-E): application to studies of environmental exposure. In: Global evidence summary. Capetown; 2017.Google Scholar
- Casey JA, Curriero FC, Cosgrove SE, Nachman KE, Schwartz BS. High-density livestock operations, crop field application of manure, and risk of community-associated methicillin-resistant Staphylococcus aureus infection in Pennsylvania. JAMA Intern Med. 2013;173:1980–90. hhttps://www.globalevidencesummit.org/workshops/new-instrument-assess-risk-bias-non-randomised-studies-exposures-robins-e-application.
- Avery RC, Wing S, Marshall SW, Schiffman SS. Odor from industrial hog farming operations and mucosal immune function in neighbors. Arch Environ Health. 2004;59:101–8.View ArticlePubMedGoogle Scholar
- Schinasi L, Horton RA, Guidry VT, Wing S, Marshall SW, Morland KB. Air pollution, lung function, and physical symptoms in communities near concentrated swine feeding operations. Epidemiology. 2011;22:208–15.View ArticlePubMedGoogle Scholar
- Wing S, Horton RA, Rose KM. Air pollution from industrial swine operations and blood pressure of neighboring residents. Environ Health Perspect. 2013;121:92–6.PubMedGoogle Scholar
- Higgins J, Sterne JA, Savović J, Page M, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials. In: Chandler J, McKenzie J, Boutron I, Welch V, editors. Cochrane methods Cochrane database of systematic reviews 2016; 2016. Issue 10 (Suppl 1) dxdoiorg/101002/14651858CD201601.Google Scholar
- Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, Guyatt G, Hooijmans C, Langendam M, Mandrioli D, et al. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92-93:611–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, Ratcliffe JM, Kraft AD, Schunemann HJ, Schwingl P, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92-93:617–29.View ArticlePubMedPubMed CentralGoogle Scholar