In this work, we have presented an overview of published systematic reviews and meta-analyses comparing laparoscopic and open surgery for colorectal cancer. We have conducted an extensive review of the literature using a highly sensitive search algorithm, and utilized the framework proposed by the Cochrane Collaboration  to synthesize 27 systematic reviews and meta-analyses published between 1994 and 2008. We have summarized the major oncologic outcome of overall survival, and provided a methodological appraisal of the reviews. Finally, we have analyzed the citation patterns of all included reviews in an attempt to understand the perceived redundancy of this body of work.
The first major finding of this overview pertains to the startling number of systematic reviews and meta-analyses identified in the literature on laparoscopic colorectal cancer surgery. There were more reviews than clinical trials, and 19 reviews were published within the span of 4 years (2005 to 2008). Given that systematic reviews are typically meant to be comprehensive in nature, this result is particularly surprising. This finding can be explained in part by the decision of certain reviewers to address only a subset of outcomes. That being said, the results shown in Table 3 would suggest that there is significant overlap between papers in terms of review purpose and outcomes. This argument is further supported by the large number of pooled analyses identified for the primary survival outcome in Figure 4 and in Additional file 2 in addition to the consistency and congruence of this result across reviews.
In addition to variation in outcomes, other authors have limited their review protocols with respect to disease subsets such as rectal cancer. This distinction may also account for a portion of the large volume of published systematic reviews, although it is unlikely to be a major factor as only four groups have focused solely on this population. More importantly, we argue that the limitation of review protocols to rectal cancer is artificial, as definitions and inclusion of rectal cancer in primary trials are highly variable. Indeed, high rectal lesions have been inconsistently defined both as rectosigmoid and rectal cancers, and this variation in terminology has been translated to different inclusion and exclusion criteria in systematic reviews. Given that high rectal cancers are typically treated in a manner that is more similar to sigmoid colon cancers, we argue that rectal cancer is best considered as part of broader reviews addressing colorectal cancer as a whole. Sensitivity analyses can then be carried out to identify outcome differences between colon and rectal cancer populations.
In addition to the above-mentioned patient population and outcomes factors, it is possible that the large number of systematic reviews be a result of a duplication of research efforts on the part of individual investigators. The analysis of citation patterns of pre-existing systematic reviews presented in Figure 5 supports this hypothesis. Indeed, we have identified that, on average, only 1.6 pre-existing reviews were referenced by individual systematic reviews. More strikingly, this corresponds to a mean ratio of cited to existing reviews of 0.16, indicating that, on average, 84% of existing systematic reviews in the literature were not found or ignored by individual review authors. This finding would support the argument that review authors either carry out poor literature evaluations or choose not to take into account pre-existing work in performing or analyzing their own reviews. Individual explanations are likely to vary, but it remains that our findings provide evidence of a duplication of research efforts in the literature.
Other examples of overlapping systematic reviews and meta-analyses can be found in the literature. In one such case, ten reviews pertaining to the use of N-acetylcysteine for the prevention of contrast-associated nephropathy were published within a 3-year span . In that particular case study, the authors documented varying quality and inconsistent recommendations. Another such analysis was published in the field of orthopedic surgery, in which different methods of anterior cruciate ligament repair were compared . A total of 11 overlapping systematic reviews were identified in this case, with varying conclusions. Interestingly, this group also identified evidence of incomplete citation of pre-existing systematic reviews.
Several academics have recently called for the registration of systematic reviews and their protocols [3, 4]. Although the Cochrane Library currently acts as a central repository of high-quality systematic reviews and meta-analyses, it is clear from our analysis that most reviewers publish their work in traditional print journals. This interpretation is supported by Moher and colleagues, who reported that approximately 2,500 new systematic reviews are published annually, of which over 90% are found in specialty journals . As such, a freely accessible registration system would have several advantages, including the early identification of pre-existing, ongoing, or unpublished reviews, the prioritization of research funding, and the enhancement of collaboration between review groups, while minimizing the possibility of research duplication. This body of information would be of particular use to practicing surgeons who would now have access to systematic reviews and meta-analyses through a single portal. Duplication of systematic reviews may be minimized with the existence of such an open-access registry, and identification of pre-existing work would likely be enhanced.
The continuous publication of new trials in time leads to the production and dissemination of new systematic reviews as a means to provide a synthesis of the literature that relies upon the latest data. For systematic reviews to be considered useful for end users, they must be up to date. In the current study, at least three groups of reviewers have provided updates of their own work in subsequent publications [10, 17, 19, 23, 29, 39]. However, these were not specifically labeled as such, a finding that may lead to confusion on the part of the reader. In contrast, all three Cochrane Reviews documented having undergone substantive amendments since their original publication, as a result of new data pertinent data in the literature [26, 30, 38]. This difference between Cochrane and non-Cochrane reviews is not surprising. Indeed, Jadad et al. have previously demonstrated that only 3% of systematic reviews published in traditional journals underwent update within 2 years of publication, compared with 38% of Cochrane Reviews . Although the timing at which systematic reviews should be updated remains controversial, it seems intuitive that, in a rapidly progressing field of healthcare such as laparoscopic surgery for colorectal cancer, existing systematic reviews should be updated frequently [87, 88]. The case for updating systematic reviews becomes particularly compelling when one considers the large number of overlapping reviews identified in this study, and when registration of systematic reviews is considered.
The methodological quality of systematic reviews included in this study was low to moderate. As indicated, the mean AMSTAR score was 5.6, although 8/27 reviews achieved a score of ≥9. It is noteworthy that all Cochrane reviews and HTA reports in this study were classified as high-quality reviews. This result is supported by existing work in the literature, which demonstrated that Cochrane reviews have greater methodological rigor than traditional print reviews .
Although many systematic reviews in this work were deemed to have a comprehensive literature search (n = 16, 59%) on AMSTAR methodology scoring, few incorporated the gray literature (n = 9, 33%). As a result, many reports fell short in their identification of published trials. Indeed, the mean ratio of cited to existing RCT reports was only 0.46 across reviews. It was only marginally better at 0.56 if one considered actual trials rather individual reports of trials which were sometimes multiple (38 reports on 23 RCTs). Only one HTA report  managed to identify all reports of existing trials. This finding is concerning in that well conducted systematic reviews are meant to be comprehensive and commonly believed to provide the highest degree of healthcare evidence .
Many factors may explain the finding of incomplete citation, including the differing patient populations and outcomes examined in each review. This is particularly relevant given the recent report by the ORBIT group on outcome reporting bias and its potential impact upon results of systematic reviews . As highlighted in the ORBIT study, it is highly important that systematic reviews avoid excluding trials on the basis of a lack of relevant data, as authors may have elected not to report a given outcome. In addition, it is also likely that the time between systematic review search completion and dissemination in electronic or print format may account for a portion of the poor citation of trials. We have attempted to avoid this form of time lag bias by allowing for a reasonable 1-year gray zone between citation of RCTs and publication (Figures 5 and 6). However, several systematic reviews had time lags to publication in excess of 2 years [19–21, 25, 32, 36] (Table 2), which may partially account for a failure to identify more recent RCTs. In addition to the above factors, it is also likely that individual search strategies were not as comprehensive as should be, in order to identify all available trials. In the current study, we have modified the highly sensitive search algorithm developed by Murray and colleagues  and have identified at least two recent trials that have not been cited in of the existing reviews [71, 73]. Moreover, at least four other reports of RCTs were cited much less frequently than other reports of trials published around the same time period [49, 54, 59, 68, 70]. In contrast, two well known RCTs were cited by almost all reviews [57, 63]. Putting together the above patterns of trials citations, we argue that the identification of trial evidence was incomplete in most retrieved systematic reviews, due at least in part to inadequate search strategies.
Our overview of all systematic reviews presenting data on oncologic outcomes reveals several important findings. First, we found no evidence of a consistent or congruent difference in overall survival between patients with colon cancer treated by laparoscopy or open surgery. Similar conclusions can be drawn from other oncologic outcomes (data not shown; Additional file 2). This result is likely to be extendable to patients with high rectal cancer as many large trials included this pathology [62, 66, 76]. However, our analysis cannot be extrapolated to those with mid or low rectal cancer, as too few trials have included these patients. Results from the large multicenter and multinational COLOR II , ACOSOG Z6051  trials will shed light onto this area of uncertainty. That being said, it is important to note that the above conclusion is limited by the lack of proper analysis of time-to-event data. Indeed, only three meta-analyses addressing overall survival presented their data in the form of HR [34, 35, 38]. Instead, many authors simply used pooled OR, which incorporate the proportion of patients alive or dead at a given point in time in each study. While this approach provides some information on survival, it is potentially biased by variable lengths of follow-up, different trial maturity, and the incomplete utilization of available data from patient censoring . Because many RCT authors do not report HR, statistical methods exist to generate such estimates from Kaplan-Meier curves [92–94]. We advocate that review groups should attempt to gather this type of data when addressing survival or other oncologic outcomes.