Evidence-based medicine provides an important framework for clinical decision making [1]. The utilization of evidence-based medicine in surgery requires a clinician to find the best available evidence and to critically appraise the validity and usefulness of the information [2]. Unfortunately, clinical evidence in the literature is of unequal quality. While well-conducted clinical trials may provide the highest level of evidence, many clinical questions are difficult to answer with trials. This is often due to side effects of interventions and various ethical dilemmas [3]. Surgical trials, in particular, face the additional challenge of clinical heterogeneity associated with varied techniques, perioperative care, and surgeon and supporting staff learning curves during the course of a study [4–6]. As a result, surgical trials have been few and far between, with surgical decision making remaining heavily influenced by a large body of observational literature.
In order to address potential confounders associated with their design, observational studies typically use statistical methods to compare study groups as well as to establish the association between intervention and outcome. Despite a variety of possible statistical manipulations, empirical work has shown that the effects of interventions in observational studies can be different in direction and magnitude when compared to that of randomized controlled trials [7, 8]. This discrepancy can be potentially attributed to the variable quality of statistical methodology used in observational studies. As a consequence, the statistical methodology can clearly influence our ability to evaluate whether confounding has been sufficiently accounted for in a given study. It is therefore important to be comprehensive and transparent with statistical reporting when publishing observational studies.
Empirical research evidence would suggest that a significant proportion of articles are flawed in the application and reporting of statistical methods [9–11]; errors could be severe enough to jeopardize the conclusion reached by the authors [12]. Many of the articles with noticeable statistical deficiencies are found in highly-referenced clinical journals [13, 14]. For instance, one study examined 100 papers in cancer journals and found that missing data may be found in 96% of the articles, with only 10% having explored the impact of such missing data on outcomes [13]. Indeed, it is known that missing data may introduce bias leading to under- and over-estimation of association between the exposure and outcome [15]. The amount of missing data also serves as a measure of study quality. Hence, it is important for the authors to provide sufficient information on missing data to enable accurate judgment of study quality. As Lang et al. have argued, such problems of poor statistical reporting concerning basic statistics are long-standing and widespread, but often go undetected [16].
In 2008, the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement was published to standardize the overall quality of reporting of observational studies [17]. The STROBE statement, however, focuses more on general quality assessment and is limited to addressing the specific statistical adjustments employed by authors. To complement the STROBE guidelines with more specific criteria, the EQUATOR (Enhancing the QUAlity and Transparency of health Research) network published the Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines to assess the quality of statistical reporting based on the type of analysis performed by authors [18].
Given that surgical decision making continues to rely heavily upon observational studies and that the validity of such work depends in large part upon adequate statistical analysis, it becomes particularly important to examine the quality and reporting of such analyses. As such, the objective of the proposed systematic review is to assess and compare the quality and reporting of statistical methods in surgical observational studies published in the highest-impact general surgical and general medical journals in 2013. More specifically, this work will adapt and utilize a tool to evaluate the quality and reporting of statistical analysis in observational studies, evaluate the risk of statistical deficiencies, compare the quality and reporting of statistical analysis in studies published on surgical topics in surgical and medical journals, and identify factors associated with high-quality reporting. This work’s primary hypothesis is that reporting of statistical methods will be generally poor for all surgical observational studies, and that reporting within the highest referenced medical journals will be superior to that published in surgical journals. The basis for this hypothesis resides with the knowledge that general medical journals tend to have much higher impact factors than surgical journals [19], while evidence suggests that higher impact factors may be associated with higher methodological quality [20].
It can be expected that this work will be significant in defining the degree of deficiencies in the quality and reporting of statistical methods in surgical observational studies, and may be used to drive improvements.