Skip to main content

The quality of research synthesis in surgery: the case of laparoscopic surgery for colorectal cancer



Several systematic reviews and meta-analyses populate the literature on the effectiveness of laparoscopic surgery for colorectal cancer. The utility of this body of work is unclear. The objective of this study was to synthesize all such systematic reviews in terms of clinical effectiveness, to appraise their quality, and to determine whether areas of duplication exist across reviews.


Systematic reviews comparing laparoscopic and open surgery for colorectal cancer were identified using a comprehensive search protocol (1991 to 2008). The primary outcome was overall survival. The methodological quality of reviews was appraised using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. Abstraction and quality appraisal was carried out by two independent reviewers. Reviews were synthesized, and outcomes were compared qualitatively. A citation analysis was carried out using simple matrices to assess the comprehensiveness of each review.


In total, 27 reviews were included; 13 reviews included only randomized controlled trials. Rectal cancer was addressed exclusively by four reviews. There was significant overlap between review purposes, populations and, outcomes. The mean AMSTAR score (out of 11) was 5.8 (95% CI: 4.6 to 7.0). Overall survival was evaluated by ten reviews, none of which found a significant difference. Three reviews provided a selective meta-analysis of time-to-event data. Previously published systematic reviews were poorly and highly selectively referenced (mean citation ratio 0.16, 95% CI: 0.093 to 0.22). Previously published trials were not comprehensively identified and cited (mean citation ratio 0.56, 95% CI: 0.46 to 0.65).


Numerous overlapping systematic reviews of laparoscopic and open surgery for colorectal cancer exist in the literature. Despite variable methods and quality, survival outcomes are congruent across reviews. A duplication of research efforts appears to exist in the literature. Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective. This works lends support to the registration and updating of systematic reviews.

Peer Review reports


Any field of active investigation in healthcare requires that the overwhelming volume of cumulative information generated by individual researchers be condensed and summarized into a usable product. This synthesis must be simple, yet comprehensive, so as to inform decisions and policies carried out by physicians and surgeons, hospital administrators, healthcare payers, funding agencies, and other end users of research and outcomes data. Systematic reviews are widely considered to be the most comprehensive and unbiased method to do so [1]. On the basis of their completeness, such reviews should remain unique in the literature and be updated frequently, rather than duplicated or compartmentalized. In this context, some authors have recently advocated for the open registration of systematic reviews [24].

Laparoscopic colorectal surgery was first described in 1991 by Fowler and White [5] and by Jacobs and colleagues [6]. This technology has since been applied to almost every disease process, whether benign or malignant, involving the colon and rectum [7]. The use of laparoscopy rather than traditional open laparotomy to treat colorectal cancer has generated tremendous controversy in the surgical literature, particularly as concerns the oncologic adequacy of this technique. Many investigators have attempted to address this issue, and have thus generated a large body of literature over the past 20 years. Published studies have included the entire spectrum of research data, ranging from small personal case series to large nationally funded multicenter randomized controlled trials (RCTs). This work presents an excellent opportunity for a case study of research synthesis and knowledge translation processes in surgical research, an area that has traditionally lacked investigative rigor [8].

Numerous systematic reviews and meta-analyses pertaining to laparoscopic surgery for colorectal cancer have been published. Informal examination of these reviews would suggest significant overlap and possible duplication. The utility of this body of work is unclear at this time. As such, we set out to examine and appraise all existing systematic reviews of laparoscopic colorectal surgery for cancer, both in terms of clinical outcomes and their relative completeness, methodological quality, and overlap.


This overview of systematic reviews was carried out using the framework for umbrella reviews described by the Cochrane Collaboration [9]. This approach consisted of identifying all existing systematic reviews and meta-analyses pertaining to laparoscopic surgery for colorectal cancer. This work was part of a larger review effort addressing both primary publications and review papers, the results of which will be presented separately. A review protocol was utilized for the project as a whole.

Criteria for considering reviews for inclusion

All inclusion and exclusion criteria were defined a priori. All systematic reviews and meta-analyses addressing laparoscopic and open surgery for colorectal cancer were included. For this purpose, all reviews were allowable if they were self-described as systematic, whether in the title, abstract, or methods of the paper. Alternatively, a citation was also allowable if the authors presented a meta-analysis of primary papers or utilized meta-analytic techniques to pool primary data. These criteria were utilized regardless of the quality or comprehensiveness of the review. The type of primary data papers included in the citations could be RCTs, observational studies, or both.

All included citations reviewed primary papers addressing the curative resection of colon and/or rectal cancer. Patients with colorectal cancer did not have to be the sole population under review. Laparoscopic resection for colorectal cancer was the intervention under study. Included reviews had to present a comparison to an open resection control group.

The primary outcome of interest was overall survival. Secondary endpoints were also considered if they were included in a review of interest: operative outcomes, short-term postoperative outcomes, oncologic surrogate outcomes, long-term oncologic outcomes, other long-term outcomes, and quality of life. A review could be included in this overview even if no data pertaining to survival was presented, as identification of review deficits was a prespecified objective of our work. Reviews addressing exclusively cost or immune function were excluded.

Search methods for identification of reviews

Systematic reviews and meta-analyses were identified as part of a broader comprehensive search strategy designed to identify primary comparative literature pertaining to laparoscopic and open surgery for colorectal cancer. The final search algorithm was devised in conjunction with an experienced information specialist from the Ottawa Hospital Library (Additional file 1). This search strategy was designed to be highly sensitive, and was modified from previously published work [10]. Six major databases were searched for relevant citations from 1991 to 2008 (Ovid MEDLINE, Ovid EMBASE, Cochrane Library, Science Citation Index Expanded, BIOSIS Preview, and BIREME LILACS). An additional 13 databases were also searched for relevant citations (Database of Abstracts of Reviews of Effectiveness, Heath Technology Assessment Database, NHS Economic Evaluation Database, NIHR Health Technology Assessment Programme, Trip Database,,, National Guidelines Clearinghouse, CMA Infobase: Clinical Practice Guidelines, NICE England, SIGN Scotland, NHMRC Australia, New Zealand Guidelines Group). The reference lists of all included citations were also screened to identify missing reviews. No language limitation was applied to the search strategy. All citation records were retrieved and downloaded electronically using Reference Manager 10 (ISI ResearchSoft, Berkeley, CA, USA), and were then de-duplicated.

Selection of reviews

All citations were first screened for inclusion by one reviewer (GM) on the basis of titles and abstracts (Figure 1). All retained citations were then retrieved in full text. Papers that could not be obtained after extensive interlibrary searching were considered missing. Papers published in languages other than English, French, or Spanish were translated in full using Google Translate (Google Inc., Mountain View, CA, USA). Full-text articles were evaluated for inclusion by one reviewer (GM). Included articles were then classified as (1) data papers or (2) review papers. Review papers were then further divided as systematic reviews/meta-analyses, narrative reviews, textbook chapters, and guidelines/position papers. Only systematic reviews/meta-analyses were considered in the current work, while all other included paper types were set aside for a separate research project. All included systematic reviews were further evaluated for inclusion by a second reviewer (SD), and disagreements between the two reviewers was resolved by discussion and consensus.

Figure 1

PRISMA diagram [95]. Note that the total number of data and review articles does not add up to 669, as 9 papers were included in both categories.

Data extraction and management

A total of 6 study characteristics and 19 prespecified outcomes of interests were extracted by two reviewers (GM, SD) onto data tables designed a priori (Table 1). All discrepancies were resolved by discussion and consensus. Relative outcomes were always recorded as a comparison of laparoscopic to open surgery. Where papers compared open to laparoscopic surgery, the appropriate reciprocal relative measure of effect and reciprocal measure of significance were calculated and recorded. The relative measures of effect were maintained in the statistical format provided by the authors of individual citations.

Table 1 List of characteristics and outcomes extracted from each review

Assessment of methodological quality of included reviews

The methodological quality of individual reviews included in this overview was assessed using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument [11]. This tool consists of 11 individual items, which were developed from pre-existing instruments, empirical evidence, and expert consensus. It has been shown to be valid, reliable, and relatively easy to use [1113]. Each item within the instrument can receive 1 point, for a possible range of AMSTAR scores of 0 to 11. The AMSTAR instrument was administered independently by two reviewers (GM, SD) and discrepant scores were resolved by discussion and consensus.

Data synthesis

The data extracted from each included systematic review was incorporated into summary tables and figures. The resulting data were synthesized in narrative form and correlated qualitatively with methodological AMSTAR scores. The congruence of pooled summary estimates for outcomes of interest across reviews was evaluated qualitatively.

As a means of evaluating the appropriateness of included reviews, a bibliographic analysis was carried out. Citation matrices [14] were generated that crosslinked individual reviews with (1) previously published systematic reviews identified in this work, and (2) published RCTs of laparoscopic vs open colorectal cancer surgery also identified in this work. This technique was used to determine whether the authors of included reviews made use of other pre-existing systematic reviews, and whether the identification of relevant RCTs was comprehensive. A 1-year lag time to publication was incorporated into the citation matrices and calculations. To account for the growing number of publications in time, the ratio of cited to total pre-existing publications was calculated for each systematic review. The mean ratios and 95% confidence interval were calculated for citation of both pre-existing systematic reviews and RCTs. Finally, citation ratios for individual systematic reviews were compared to AMSTAR scores and 5-year impact factors (where available) [15] using Pearson correlation coefficients.

All calculations and statistical analyses were performed using Microsoft Excel 2010 (Microsoft Corporation, Richmond, WA, USA) and SAS V. 9.2 (SAS Institute Inc., Cary, NC, USA).


Description of included reviews

A total of 5,800 single citation records were screened for inclusion (Figure 1). After applying a 2-step screening process, a total of 27 full-text systematic reviews and/or meta-analyses [10, 1641] were included in this overview. Not included in this final tally was one Chinese language review, which could not be translated [42]. Similarly, another systematic review was excluded, as it focused solely on the methodological quality of RCTs identified in a Cochrane Review by the same author [43]. Finally, two more reviews were not considered as they addressed cost outcomes [44] and hand-assisted vs standard laparoscopic colorectal surgery [45], respectively. Other primary data papers and reviews identified in Figure 1 have also been included, and will form the basis for a separate publication. The full list of citations is available from the authors upon request.

The characteristics of all included reviews are provided in Table 2. Included reviews were published between 1994 and 2008, of which 19/27 (70%) were published in 2005 or later. Three groups published updates of their own reviews [10, 17, 19, 23, 29, 39]. Because these were not labeled as updates in the titles or abstracts, they were included as independent reviews. A majority of reviews (n = 18, 67%) were published in traditional print journals, while the remainder comprised commissioned reports (n = 4, 15%) [16, 17, 23, 24], Cochrane Reviews (n = 3, 11%) [26, 30, 38], and Health Technology Assessment (HTA) report (n = 2, 7.4%) [10, 18]. The mean 5-year impact factor associated with published reports was 4.02 (n = 20, 95% CI: 3.07 to 4.96). These originated in 12 countries, with Europe (n = 14, 52%) and Australia (n = 5, 19%) accounting for a majority of papers. Five reviews were presented in non-English languages, including French (n = 2), Spanish (n = 2), and Norwegian (n = 1).

Table 2 Characteristics of included reviews

Among included reviews, four (15%) addressed exclusively rectal cancer [27, 30, 31, 41], one (3.7%) reviewed only colon cancer [25], whereas the remainder were less specific and identified 'colorectal cancer' as their population of choice (Tables 2 and 3). This descriptor led to contradictions in certain papers, as the authors sometimes specifically excluded rectal cancer, but then went on to include papers with 'rectosigmoid cancers' or 'anterior resections' [17, 19, 34]. A total of four (15%) reviews were wide in scope and addressed the laparoscopic treatment of all colorectal pathologies, of which colorectal cancer was a subset of patients [16, 20, 21, 33]. All reviews sought to find studies comparing laparoscopic and open radical resection for cancer. One early paper found no controlled study, and was thus limited to case series of laparoscopic surgery for colorectal cancer [16]. Right hemicolectomy, left hemicolectomy, and sigmoid resection were the most commonly included types of colonic resections. All publications not limited exclusively to rectal cancer included those three procedures. In contrast, studies including transverse colectomies were frequently excluded by review authors [17, 19, 22, 32].

Table 3 Summary of reviewed populations and outcomes

All reviews except one set out to include RCTs as part of their analyses (Table 2). One group chose to focus solely upon observational studies [36]. A total of 13 (48%) reviews allowed only RCTs as part of their inclusion criteria. Of these, 77% were published in 2006 or later. One such group selected only four larger RCTs, and carried out a meta-analysis of individual patient data [34]. Two reviews whose last literature searches were in 1993 [16] and 1997 [20], respectively, were unable to identify any published RCTs. As a result, both groups presented only observational studies.

Table 3 provides an overview of the purposes, disease processes, and outcomes addressed by each systematic review. The self-described purposes were highly comparable, with most papers choosing to address issues of efficacy or effectiveness in broad terms. Long-term oncologic outcomes were sought by 19/27 reviewers (70%), while oncologic surrogates were found in 22 (81%) reviews. Similarly, operative and short-term postoperative outcomes were analyzed in 74% (n = 20) and 78% (n = 21) of reviews, respectively. A total of 13 (48%) reviews addressed all 4 outcomes types, while 18 (67%) included at least 3 of the 4. Table 3 reveals significant overlap in study purposes and outcomes. In contrast, long-term operative outcomes (n = 4, 15%) and perioperative quality of life (n = 6, 22%) were much less frequently included in the current group of systematic reviews.

Methodological quality

The included systematic reviews were generally of low to moderate quality. The mean AMSTAR methodological quality score was 5.8 (95% CI: 4.6 to 7.0). A total of eight reviews (30%) achieved a score of 9 or greater, and can be considered of high methodological quality. The quality of reviews appears to have improved modestly in time (Figure 2), with all but one high-quality review having been published in 2005 or later. Among high-quality reviews, three were published in traditional journals [32, 35, 39], three were Cochrane Reviews [26, 30, 38], and two were HTA reports [10, 18].

Figure 2

Assessment of Multiple Systematic Reviews (AMSTAR) methodological quality scores against time. Red circles represent Cochrane reviews and Health Technology Assessment (HTA) reports.

The composite of individual items within the AMSTAR score is presented in Figure 3. Most review groups defined their research question and inclusion criteria a priori (20/27, 74%), and documented the evaluation of the scientific quality of included studies (20/27, 74%). Similarly, 70% of systematic reviews used appropriate methods to combine individual study results (n = 19), and can be considered to have reached appropriate conclusions that reflected the methodological quality assessment of their selected papers (n = 19). In contrast, few groups addressed the possibility of publication bias (7/27, 26%), provided evidence of a duplicate study selection and data extraction process (10/27, 37%), listed all included and excluded studies (10/27, 37%), or avoided limiting their search or inclusion of studies on the basis of publication status (9/27, 33%).

Figure 3

Methodological quality of included reviews on the basis of individual Assessment of Multiple Systematic Reviews (AMSTAR) items.

Synthesis of reviewed outcomes

The primary outcome of overall survival was pooled quantitatively in ten reviews (Figure 4 and Additional file 2). Most authors presented this data as a dichotomous outcome (alive/dead) at maximum follow-up. After meta-analysis, this data was presented as odds ratios (OR), risk ratios (RR), and Stouffer's composite Z (ZC). Two groups presented this outcome as time-to-event data by pooling hazard ratios (HR), using methods relying on the estimation of HR from Kaplan-Meier curves. Finally, another group also presented a HR that was derived from an individual patient data meta-analysis. No significant difference in overall survival was found between laparoscopic and open surgery for colorectal cancer across all meta-analytic comparisons. The direction of effect for all analyses favored laparoscopy, except for two which were conducted on observational studies and using a subset of studies with independent patient data [18, 34].

Figure 4

Synthesis of overall survival across systematic reviews.

All other pooled outcomes, except for the total number of harvested lymph nodes, yielded comparable non-significant results. These findings are not presented in the main text, but are available in Additional file 2.

Bibliographic analysis

The pattern of citations of previously published systematic reviews is presented in Figure 5. In total, nine (33%) reviews did not cite any pre-existing work. The maximum number of cited existing systematic reviews was five, which was found in two reviews [29, 40]. All other studies cited zero to four pre-existing systematic reviews. The mean number of cited existing reviews was 1.6 (95% CI: 1.0 to 2.2). The mean ratio of cited to total existing systematic reviews was 0.16 (95% CI: 0.093 to 0.22), ranging from 0 to 0.5. All three Cochrane Reviews (0, 0.048 to 0.091) and the more recent HTA report (0.18) had citation ratios that were less than 0.2. There was no correlation between the citation ratio and the AMSTAR score (r = 0.047) or the journal's 5-year impact factor (r = -0.099) for individual reviews. In total, 13/26 pre-existing reviews were cited at least once, with 5 of these accounting for 71% of all citations (29/41) [1719, 22, 26]. Figure 5 shows that five reviews were cited disproportionately more frequently, and that all five reviews were published in the earlier portion of the literature review.

Figure 5

Citation of previously published systematic reviews.

The current overview protocol identified 38 publications pertaining to 23 individual RCTs comparing laparoscopic and open surgery for colorectal cancer [4683]. The patterns of citations of these RCTs are presented in Figure 6. Two systematic reviews did not cite any RCTs. In the case of Bernard et al. [16], there were no RCTs yet published in the literature. In the case of Yong et al. [20], a total of ten RCT publications could have been cited. The mean ratio of cited to total published RCT reports was 0.45 (95% CI: 0.35 to 0.54), ranging from 0 to 1. Excluding Abraham et al. [36], which sought to analyze only observational studies, yielded a comparable mean ratio of 0.46 (95% CI: 0.36 to 0.56). Given that certain RCTs were published over multiple papers, the ratio of cited to total published RCT reports was recalculated using each trial as the denominator rather than individual papers. This analysis yielded a mean ratio of 0.56 (95% CI: 0.46, 0.65). A total of only four reviews identified at least 75% of RCT publications, of which two were Cochrane Reviews and one was an HTA report [10, 26, 28, 38]. Reviews that selected rectal cancer as their sole patient population had generally low citation ratios, ranging from 0.13 to 0.39 [27, 30, 31, 40]. Among all RCT publications, two were cited disproportionately more frequently than others. Indeed, the Barcelona trial by Lacy et al. [57] and the Clinical Outcomes of Surgical Therapy (COST) trial [63] were both cited by 90% of systematic reviews. Finally, the correlation between the citation ratios and the AMSTAR scores (r = 0.43), and between the ratios and the journal's 5-year impact factors were moderate at best (r = 0.46).

Figure 6

Citation of previously published randomized controlled trials.


In this work, we have presented an overview of published systematic reviews and meta-analyses comparing laparoscopic and open surgery for colorectal cancer. We have conducted an extensive review of the literature using a highly sensitive search algorithm, and utilized the framework proposed by the Cochrane Collaboration [9] to synthesize 27 systematic reviews and meta-analyses published between 1994 and 2008. We have summarized the major oncologic outcome of overall survival, and provided a methodological appraisal of the reviews. Finally, we have analyzed the citation patterns of all included reviews in an attempt to understand the perceived redundancy of this body of work.

The first major finding of this overview pertains to the startling number of systematic reviews and meta-analyses identified in the literature on laparoscopic colorectal cancer surgery. There were more reviews than clinical trials, and 19 reviews were published within the span of 4 years (2005 to 2008). Given that systematic reviews are typically meant to be comprehensive in nature, this result is particularly surprising. This finding can be explained in part by the decision of certain reviewers to address only a subset of outcomes. That being said, the results shown in Table 3 would suggest that there is significant overlap between papers in terms of review purpose and outcomes. This argument is further supported by the large number of pooled analyses identified for the primary survival outcome in Figure 4 and in Additional file 2 in addition to the consistency and congruence of this result across reviews.

In addition to variation in outcomes, other authors have limited their review protocols with respect to disease subsets such as rectal cancer. This distinction may also account for a portion of the large volume of published systematic reviews, although it is unlikely to be a major factor as only four groups have focused solely on this population. More importantly, we argue that the limitation of review protocols to rectal cancer is artificial, as definitions and inclusion of rectal cancer in primary trials are highly variable. Indeed, high rectal lesions have been inconsistently defined both as rectosigmoid and rectal cancers, and this variation in terminology has been translated to different inclusion and exclusion criteria in systematic reviews. Given that high rectal cancers are typically treated in a manner that is more similar to sigmoid colon cancers, we argue that rectal cancer is best considered as part of broader reviews addressing colorectal cancer as a whole. Sensitivity analyses can then be carried out to identify outcome differences between colon and rectal cancer populations.

In addition to the above-mentioned patient population and outcomes factors, it is possible that the large number of systematic reviews be a result of a duplication of research efforts on the part of individual investigators. The analysis of citation patterns of pre-existing systematic reviews presented in Figure 5 supports this hypothesis. Indeed, we have identified that, on average, only 1.6 pre-existing reviews were referenced by individual systematic reviews. More strikingly, this corresponds to a mean ratio of cited to existing reviews of 0.16, indicating that, on average, 84% of existing systematic reviews in the literature were not found or ignored by individual review authors. This finding would support the argument that review authors either carry out poor literature evaluations or choose not to take into account pre-existing work in performing or analyzing their own reviews. Individual explanations are likely to vary, but it remains that our findings provide evidence of a duplication of research efforts in the literature.

Other examples of overlapping systematic reviews and meta-analyses can be found in the literature. In one such case, ten reviews pertaining to the use of N-acetylcysteine for the prevention of contrast-associated nephropathy were published within a 3-year span [84]. In that particular case study, the authors documented varying quality and inconsistent recommendations. Another such analysis was published in the field of orthopedic surgery, in which different methods of anterior cruciate ligament repair were compared [85]. A total of 11 overlapping systematic reviews were identified in this case, with varying conclusions. Interestingly, this group also identified evidence of incomplete citation of pre-existing systematic reviews.

Several academics have recently called for the registration of systematic reviews and their protocols [3, 4]. Although the Cochrane Library currently acts as a central repository of high-quality systematic reviews and meta-analyses, it is clear from our analysis that most reviewers publish their work in traditional print journals. This interpretation is supported by Moher and colleagues, who reported that approximately 2,500 new systematic reviews are published annually, of which over 90% are found in specialty journals [2]. As such, a freely accessible registration system would have several advantages, including the early identification of pre-existing, ongoing, or unpublished reviews, the prioritization of research funding, and the enhancement of collaboration between review groups, while minimizing the possibility of research duplication. This body of information would be of particular use to practicing surgeons who would now have access to systematic reviews and meta-analyses through a single portal. Duplication of systematic reviews may be minimized with the existence of such an open-access registry, and identification of pre-existing work would likely be enhanced.

The continuous publication of new trials in time leads to the production and dissemination of new systematic reviews as a means to provide a synthesis of the literature that relies upon the latest data. For systematic reviews to be considered useful for end users, they must be up to date. In the current study, at least three groups of reviewers have provided updates of their own work in subsequent publications [10, 17, 19, 23, 29, 39]. However, these were not specifically labeled as such, a finding that may lead to confusion on the part of the reader. In contrast, all three Cochrane Reviews documented having undergone substantive amendments since their original publication, as a result of new data pertinent data in the literature [26, 30, 38]. This difference between Cochrane and non-Cochrane reviews is not surprising. Indeed, Jadad et al. have previously demonstrated that only 3% of systematic reviews published in traditional journals underwent update within 2 years of publication, compared with 38% of Cochrane Reviews [86]. Although the timing at which systematic reviews should be updated remains controversial, it seems intuitive that, in a rapidly progressing field of healthcare such as laparoscopic surgery for colorectal cancer, existing systematic reviews should be updated frequently [87, 88]. The case for updating systematic reviews becomes particularly compelling when one considers the large number of overlapping reviews identified in this study, and when registration of systematic reviews is considered.

The methodological quality of systematic reviews included in this study was low to moderate. As indicated, the mean AMSTAR score was 5.6, although 8/27 reviews achieved a score of ≥9. It is noteworthy that all Cochrane reviews and HTA reports in this study were classified as high-quality reviews. This result is supported by existing work in the literature, which demonstrated that Cochrane reviews have greater methodological rigor than traditional print reviews [86].

Although many systematic reviews in this work were deemed to have a comprehensive literature search (n = 16, 59%) on AMSTAR methodology scoring, few incorporated the gray literature (n = 9, 33%). As a result, many reports fell short in their identification of published trials. Indeed, the mean ratio of cited to existing RCT reports was only 0.46 across reviews. It was only marginally better at 0.56 if one considered actual trials rather individual reports of trials which were sometimes multiple (38 reports on 23 RCTs). Only one HTA report [10] managed to identify all reports of existing trials. This finding is concerning in that well conducted systematic reviews are meant to be comprehensive and commonly believed to provide the highest degree of healthcare evidence [4].

Many factors may explain the finding of incomplete citation, including the differing patient populations and outcomes examined in each review. This is particularly relevant given the recent report by the ORBIT group on outcome reporting bias and its potential impact upon results of systematic reviews [89]. As highlighted in the ORBIT study, it is highly important that systematic reviews avoid excluding trials on the basis of a lack of relevant data, as authors may have elected not to report a given outcome. In addition, it is also likely that the time between systematic review search completion and dissemination in electronic or print format may account for a portion of the poor citation of trials. We have attempted to avoid this form of time lag bias by allowing for a reasonable 1-year gray zone between citation of RCTs and publication (Figures 5 and 6). However, several systematic reviews had time lags to publication in excess of 2 years [1921, 25, 32, 36] (Table 2), which may partially account for a failure to identify more recent RCTs. In addition to the above factors, it is also likely that individual search strategies were not as comprehensive as should be, in order to identify all available trials. In the current study, we have modified the highly sensitive search algorithm developed by Murray and colleagues [10] and have identified at least two recent trials that have not been cited in of the existing reviews [71, 73]. Moreover, at least four other reports of RCTs were cited much less frequently than other reports of trials published around the same time period [49, 54, 59, 68, 70]. In contrast, two well known RCTs were cited by almost all reviews [57, 63]. Putting together the above patterns of trials citations, we argue that the identification of trial evidence was incomplete in most retrieved systematic reviews, due at least in part to inadequate search strategies.

Our overview of all systematic reviews presenting data on oncologic outcomes reveals several important findings. First, we found no evidence of a consistent or congruent difference in overall survival between patients with colon cancer treated by laparoscopy or open surgery. Similar conclusions can be drawn from other oncologic outcomes (data not shown; Additional file 2). This result is likely to be extendable to patients with high rectal cancer as many large trials included this pathology [62, 66, 76]. However, our analysis cannot be extrapolated to those with mid or low rectal cancer, as too few trials have included these patients. Results from the large multicenter and multinational COLOR II [90], ACOSOG Z6051 [91] trials will shed light onto this area of uncertainty. That being said, it is important to note that the above conclusion is limited by the lack of proper analysis of time-to-event data. Indeed, only three meta-analyses addressing overall survival presented their data in the form of HR [34, 35, 38]. Instead, many authors simply used pooled OR, which incorporate the proportion of patients alive or dead at a given point in time in each study. While this approach provides some information on survival, it is potentially biased by variable lengths of follow-up, different trial maturity, and the incomplete utilization of available data from patient censoring [38]. Because many RCT authors do not report HR, statistical methods exist to generate such estimates from Kaplan-Meier curves [9294]. We advocate that review groups should attempt to gather this type of data when addressing survival or other oncologic outcomes.


A large number of overlapping systematic reviews and meta-analyses comparing laparoscopic and open surgery for colorectal cancer can be identified in the literature. The methodological quality of systematic reviews is generally low to moderate, as evidenced by the incomplete identification of published trials. On the whole, Cochrane Reviews and Health Technology Assessment reports demonstrate higher quality indices than most traditional print reviews. Survival outcomes are inconsistently reported and time-to-event data are infrequently included in pooled estimates. That being said, all pooled estimates of overall survival comparing laparoscopic and open surgery for colorectal cancer are congruent and demonstrate no significant difference. There appears to be evidence of duplication of research efforts among review groups, as evidenced by overlapping review purposes, populations, and outcomes, as well as by the poor citation of pre-existing systematic reviews. Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective. This works lends support to the registration and updating of systematic reviews.


  1. 1.

    Mulrow CD: Rationale for systematic reviews. BMJ. 1994, 309: 597-599. 10.1136/bmj.309.6954.597.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007, 4: e78-10.1371/journal.pmed.0040078.

    Article  PubMed  PubMed Central  Google Scholar 

  3. 3.

    Straus S, Moher D: Registering systematic reviews. CMAJ. 2010, 182: 13-14. 10.1503/cmaj.081849.

    Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L: An international registry of systematic-review protocols. Lancet. 2011, 377: 108-109. 10.1016/S0140-6736(10)60903-8.

    Article  PubMed  Google Scholar 

  5. 5.

    Fowler DL, White A: Laparoscopy-assisted sigmoid resection. Surg Laparosc Endosc. 1991, 1: 183-188.

    CAS  Article  PubMed  Google Scholar 

  6. 6.

    Jacobs M, Verdeja JC, Goldstein HS: Minimally invasive colon resection (laparoscopic colectomy). Surg Laparosc Endosc. 1991, 1: 144-150.

    CAS  PubMed  Google Scholar 

  7. 7.

    Martel G, Boushey RP: Laparoscopic colon surgery: past, present and future. Surg Clin N Am. 2006, 86: 867-897. 10.1016/j.suc.2006.05.006.

    Article  PubMed  Google Scholar 

  8. 8.

    Barkun JS, Aronson JK, Feldman LS, Maddern GJ, Strasberg SM: Surgical innovation 1: evaluation and stages of surgical innovation. Lancet. 2009, 374: 1089-1096. 10.1016/S0140-6736(09)61083-7.

    Article  PubMed  Google Scholar 

  9. 9.

    Becker LA, Oxman AD: Chapter 22: Overviews of reviews. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. 2008, Chichester, UK: The Cochrane Collaboration and John Wiley & Sons Ltd, 607-631.

    Chapter  Google Scholar 

  10. 10.

    Murray A, Lourenco T, de Verteuil R, Hernandez R, Fraser C, McKinley A, Krukowski Z, Vale L, Grant A: Clinical effectiveness and cost-effectiveness of laparoscopic surgery for colorectal cancer: systematic reviews and economic evaluation. Health Technol Assess. 2006, 10: 1-141.

    CAS  Article  PubMed  Google Scholar 

  11. 11.

    Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM: External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007, 2: e1350-10.1371/journal.pone.0001350.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw JM, Henry DA, Boers M: AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009, 62: 1013-1020. 10.1016/j.jclinepi.2008.10.009.

    Article  PubMed  Google Scholar 

  14. 14.

    Fergusson D, Glass KC, Hutton B, Shapiro S: Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding?. Clinical Trials. 2005, 2: 218-232. 10.1191/1740774505cn085oa.

    Article  PubMed  Google Scholar 

  15. 15.

    ISI Web of Knowledge: Journal Citation Reports 2009. []

  16. 16.

    Bernard A, Durieux P: Évaluation des méthodes coélioscopiques en chirurgie digestive. 1994, Paris, France: Agence Nationale pour le Développement de l'Évaluation Médicale

    Google Scholar 

  17. 17.

    Chapman A, DeNichilo D, Babidge W, Madddern G, Hewett P, Levitt M, Sheiner H, Woods R: Systematic review of laparoscopic-assisted resection of colorectal malignancies. 2000, ASERNIP-S Report No. 8 Adelaide, Australia: ASERNIP-S

    Google Scholar 

  18. 18.

    Vardulaki KA, Bennett-Lloyd BD, Parfitt J, Normond C, Paisley S, Darzi A, Reeves BC: A systematic review of the effectiveness and cost-effectiveness of laparoscopic surgery for colorectal cancer. 2000, London, UK: National Institute for Health and Clinical Excellence

    Google Scholar 

  19. 19.

    Chapman AE, Levitt MD, Hewett P, Woods R, Sheiner H, Maddern GJ: Laparoscopic-assisted resection of colorectal malignancies: a systematic review. Ann Surg. 2001, 234: 590-606. 10.1097/00000658-200111000-00003.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Yong L, Deane M, Monson JRT, Darzi A: Systematic review of laparoscopic surgery for colorectal malignancy. Surg Endosc. 2001, 15: 1431-1439.

    CAS  Article  PubMed  Google Scholar 

  21. 21.

    Korolija D, Tadic S, Simic D: Extent of oncological resection in laparoscopic vs. open colorectal surgery: meta-analysis. Langenbecks Arch Surg. 2003, 387: 366-371.

    CAS  PubMed  Google Scholar 

  22. 22.

    Abraham NS, Young JM, Solomon MJ: Meta-analysis of short-term outcomes after laparoscopic resection for colorectal cancer. Br J Surg. 2004, 91: 1111-1124. 10.1002/bjs.4640.

    CAS  Article  PubMed  Google Scholar 

  23. 23.

    Reza MM, Blasco JA: Eficacia y seguridad de la cirugía laparoscópica vs cirugía abierta en el cancer colorrectal. 2004, Madrid, Spain: Unidad de Evaluacíon de Tecnologías Sanitarias (UETS), Área de Investigacíon y Estudios Sanitarios Agencia Laín Entralgo

    Google Scholar 

  24. 24.

    Poutignat N, Bataille N, Obrecht O, Zaleski ID: Comparison of laparosopic and open surgery in colorectal cancer. 2005, Saint-Denis La Plaine, France: Haute Autorité de Santé

    Google Scholar 

  25. 25.

    Manterola C, Pineda V, Vial M: Reseccíon abierta frente a laparoscópica en el cancer de colon no complicado. Revisión sistemática. Cir Esp. 2005, 78: 28-33.

    Article  PubMed  Google Scholar 

  26. 26.

    Schwenk W, Haase O, Neudecker JJ, Müller JM: Short term benefits for laparoscopic colorectal resection. Cochrane Database Syst Rev. 2005, 3: CD003145-

    PubMed  Google Scholar 

  27. 27.

    Aziz O, Constantinides V, Tekkis PP, Athanasiou T, Purkayastha S, Paraskeva P, Darzi AW, Heriot AG: Laparoscopic versus open surgery for rectal cancer: a meta-analysis. Ann Surg Oncol. 2006, 13: 413-424. 10.1245/ASO.2006.05.045.

    Article  PubMed  Google Scholar 

  28. 28.

    Tjandra JJ, Chan MKY: Systematic review on the short-term outcome of laparoscopic resection for colon and rectosigmoid cancer. Colorectal Dis. 2006, 8: 375-388. 10.1111/j.1463-1318.2006.00974.x.

    CAS  Article  PubMed  Google Scholar 

  29. 29.

    Reza MM, Blasco JA, Andradas E, Centero R, Mayol J: Systematic review of laparoscopic versus open surgery for colorectal cancer. Br J Surg. 2006, 93: 921-928. 10.1002/bjs.5430.

    CAS  Article  PubMed  Google Scholar 

  30. 30.

    Breukink S, Pierie JP, Wiggers T: Laparoscopic versus open total mesorectal excision for rectal cancer. Cochrane Database Syst Rev. 2006, 4: CD005200-

    PubMed  Google Scholar 

  31. 31.

    Gao F, Cao YF, Chen LS: Meta-analysis of short-term outcomes after laparoscopic resection for rectal cancer. Int J Colorectal Dis. 2006, 21: 652-656. 10.1007/s00384-005-0079-0.

    Article  PubMed  Google Scholar 

  32. 32.

    Kahnamoui K, Cadeddu M, Farrokhyar F, Anvari M: Laparoscopic surgery for colon cancer: a systematic review. Can J Surg. 2007, 50: 48-57.

    PubMed  PubMed Central  Google Scholar 

  33. 33.

    Noel JK, Fahrbach K, Estok R, Cella C, Frame D, Linz H, Cima RR, Dozois EJ, Senagore AJ: Minimally invasive colorectal resection outcomes: short-term comparison with open procedures. J Am Coll Surg. 2007, 204: 291-307. 10.1016/j.jamcollsurg.2006.10.002.

    Article  PubMed  Google Scholar 

  34. 34.

    Bonjer HC, Hop WC, Nelson H, Sargent DJ, Lacy AM, Castells A, Guillou PJ, Thorpe H, Brown J, Delgado S, Kuhri E, Haglind E, Påhlman L, Transatlantic Laparoscopically Assisted vs Open Colectomy Trials Study Group: Laparoscopically assisted vs open colectomy for colon cancer: meta-analysis. Arch Surg. 2007, 142: 298-303.

    Article  PubMed  Google Scholar 

  35. 35.

    Jackson TD, Kaplan GG, Arena G, Page JH, Rogers SO: Laparoscopic versus open resection for colorectal cancer: a metaanalysis of oncologic outcomes. J Am Coll Surg. 2007, 204: 439-446. 10.1016/j.jamcollsurg.2006.12.008.

    Article  PubMed  Google Scholar 

  36. 36.

    Abraham NS, Byrne CM, Young JM, Solomon MJ: Meta-analysis of non-randomized comparative studies of the short-term outcomes of laparoscopic resection for colorectal cancer. ANZ J Surg. 2007, 77: 508-516. 10.1111/j.1445-2197.2007.04141.x.

    Article  PubMed  Google Scholar 

  37. 37.

    Kuhry E, Sætnan E, Græslie H, Gaupset R: Laparoskopi ved kolorektal kreft. Tidsskr Nor Lægeforen. 2007, 127: 2946-2949.

    PubMed  Google Scholar 

  38. 38.

    Kuhry E, Schwenk W, Gaupset R, Romild U, Bonjer HJ: Long-term results of laparoscopic colorectal cancer resection. Cochrane Database Syst Rev. 2008, 2: CD003432-

    PubMed  Google Scholar 

  39. 39.

    Lorenco T, Murray A, Grant A, McKinley A, Krukowski Z, Vale L: Laparoscopic surgery for colorectal cancer: safe and effective? - A systematic review. Surg Endosc. 2008, 22: 1146-1160. 10.1007/s00464-007-9686-x.

    Article  Google Scholar 

  40. 40.

    Anderson C, Uman G, Pigazzi A: Oncologic outcomes of laparoscopic surgery for colorectal cancer: a systematic review and meta-analysis of the literature. Eur J Surg Oncol. 2008, 34: 1135-1142. 10.1016/j.ejso.2007.11.015.

    CAS  Article  PubMed  Google Scholar 

  41. 41.

    Liang Y, Li G, Chen P, Yu J: Laparoscopic versus open colorectal resection for cancer: a meta-analysis of results of randomized controlled trials on recurrence. Eur J Surg Oncol. 2008, 34: 1217-1224. 10.1016/j.ejso.2007.11.004.

    CAS  Article  PubMed  Google Scholar 

  42. 42.

    Li J, Ding K, Zhang S: Meta-analysis of short-term efficacy and safety after laparoscopic resection for colorectal cancer. Natl Med J China. 2006, 86: 2485-2490.

    Google Scholar 

  43. 43.

    Schwenk W, Haase O, Günther N, Neudecker J: Methodological quality of randomised controlled trials comparing short-term results of laparoscopic and conventional colorectal resection. Int J Colorectal Dis. 2007, 22: 1369-1376. 10.1007/s00384-007-0318-7.

    Article  PubMed  Google Scholar 

  44. 44.

    Aalbers AG, Biere SS, van Berge Henegouwen MI, Bemelman WA: Hand-assisted or laparoscopic-assisted approach in colorectal surgery: a systematic review and meta-analysis. Surg Endosc. 2008, 22: 1769-1780. 10.1007/s00464-008-9857-4.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  45. 45.

    Dowson HM, Huang A, Soon Y, Gage H, Lovell DP, Rockall TA: Systematic review of the costs of laparoscopic colorectal surgery. Dis Colon Rectum. 2007, 50: 908-919. 10.1007/s10350-007-0234-5.

    Article  PubMed  Google Scholar 

  46. 46.

    Lacy AM, Garcia-Valdecasas JC, Pique JM, Delgado S, Campo E, Bordas JM, Taura P, Grande L, Fuster J, Pacheco JL, Visa J: Short-term outcome analysis of a randomized study comparing laparoscopic vs open colectomy for colon cancer. Surg Endosc. 1995, 9: 1101-1105.

    CAS  PubMed  Google Scholar 

  47. 47.

    Stage JG, Schulze S, Moller P, Overgaard H, Andersen M, Rebsdorf-Pedersen VB, Nielsen HJ: Prospective randomized study of laparoscopic versus open colonic resection for adenocarcinoma. Br J Surg. 1997, 84: 391-339. 10.1002/bjs.1800840339.

    CAS  Article  PubMed  Google Scholar 

  48. 48.

    Schwenk W, Böhm B, Haase O, Junghans T, Müller JM: Laparoscopic versus conventional colorectal resection: a prospective randomized study of postoperative ileus and early postoperative feeding. Langenbecks Arch Surg. 1998, 383: 49-55. 10.1007/s004230050091.

    CAS  Article  PubMed  Google Scholar 

  49. 49.

    Schwenk W, Bohm B, Muller JM: Influence of laparoscopic or conventional colorectal resection on postoperative quality of life. Zentralbl Chir. 1998, 123: 483-490.

    CAS  PubMed  Google Scholar 

  50. 50.

    Milsom JW, Böhm B, Hammerhofer KA, Fazio V, Steiger E, Elson P: A prospective, randomized trial comparing laparoscopic versus conventional techniques in colorectal cancer surgery: a preliminary report. J Am Coll Surg. 1998, 187: 46-54. 10.1016/S1072-7515(98)00132-X.

    CAS  Article  PubMed  Google Scholar 

  51. 51.

    Lacy AM, Delgado S, Garcia-Valdecasas JC, Castells A, Pique JM, Grande L, Fuster J, Targarona EM, Pera M, Visa J: Port site metastases and recurrence after laparoscopic colectomy: a randomized trial. Surg Endosc. 1998, 12: 1039-1042. 10.1007/s004649900776.

    CAS  Article  PubMed  Google Scholar 

  52. 52.

    Schwenk W, Bohm B, Muller JM: Postoperative pain and fatigue after laparoscopic or conventional colorectal resections. Surg Endosc. 1998, 12: 1131-1136. 10.1007/s004649900799.

    CAS  Article  PubMed  Google Scholar 

  53. 53.

    Schwenk W, Bohm B, Witt C, Junghans T, Grundel K, Muller JM: Pulmonary function following laparoscopic or conventional colorectal resection: a randomized controlled evaluation. Arch Surg. 1999, 134: 6-12. 10.1001/archsurg.134.1.6.

    CAS  Article  PubMed  Google Scholar 

  54. 54.

    Delgado S, Lacy AM, Garcia-Valdecasas JC, Balague C, Pera M, Salvador L, Momblan D, Visa J: Could age be an indicator for laparoscopic colectomy in colorectal cancer. Surg Endosc. 2000, 14: 22-26. 10.1007/s004649900004.

    CAS  Article  PubMed  Google Scholar 

  55. 55.

    Curet MJ, Putrakul K, Pitcher DE, Josloff RK, Zucker KA: Laparoscopically assisted colon resection for colon carcinoma: prospective results and long-term outcome. Surg Endosc. 2000, 14: 1062-1066. 10.1007/s004640000092.

    CAS  Article  PubMed  Google Scholar 

  56. 56.

    Weeks JC, Nelson H, Gelber S, Sargent D, Schroeder G: Short-term quality-of-life outcomes following laparoscopic-assisted colectomy vs open colectomy for colon cancer: a randomized trial. JAMA. 2002, 287: 321-328. 10.1001/jama.287.3.321.

    Article  PubMed  Google Scholar 

  57. 57.

    Lacy AM, Garcia-Valdecasas JC, Delgado S, Castells A, Taura P, Pique JM, Visa J: Laparoscopy-assisted colectomy versus open colectomy for treatment of non-metastatic colon cancer: a randomized trial. Lancet. 2002, 359: 2224-2229. 10.1016/S0140-6736(02)09290-5.

    Article  PubMed  Google Scholar 

  58. 58.

    Winslow ER, Fleshman JW, Birnbaum EH, Brunt LM: Wound complications of laparoscopic vs open colectomy. Surg Endosc. 2002, 16: 1420-1425. 10.1007/s00464-002-8837-3.

    CAS  Article  PubMed  Google Scholar 

  59. 59.

    Quah HM, Jayne DG, Eu KW, Seow-Choen F: Bladder and sexual dysfunction following laparoscopically assisted and conventional open mesorectal resection for cancer. Br J Surg. 2002, 89: 1551-1556. 10.1046/j.1365-2168.2002.02275.x.

    CAS  Article  PubMed  Google Scholar 

  60. 60.

    Hasegawa H, Kabeshima Y, Watanabe M, Yamamoto S, Kitajima M: Randomized controlled trial of laparoscopic versus open colectomy for advanced colorectal cancer. Surg Endosc. 2003, 17: 636-640. 10.1007/s00464-002-8516-4.

    CAS  Article  PubMed  Google Scholar 

  61. 61.

    Araujo SE, da Silva e Sousa AH, de Campos FG, Habr-Gama A, Dumarco RB, Caravatto PP, Nahas SC, da Silva J, Kiss DR, Gama-Rodrigues JJ: Conventional approach x laparoscopic abdominoperineal resection for rectal cancer treatment after neoadjuvant chemoradiation: results of a prospective randomized trial. Rev Hosp Clin Fac Med Sao Paulo. 2003, 58: 133-140. 10.1590/S0041-87812003000300002.

    Article  PubMed  Google Scholar 

  62. 62.

    Leung KL, Kwok SP, Lam SC, Lee JF, Yiu RY, Ng SS, Lai PB, Lau WY: Laparoscopic resection of rectosigmoid carcinoma: prospective randomized trial. Lancet. 2004, 363: 1187-1192. 10.1016/S0140-6736(04)15947-3.

    Article  PubMed  Google Scholar 

  63. 63.

    Clinical Outcomes of Surgical Therapy Study Group: A comparison of laparoscopically assisted and open colectomy for colon cancer. N Engl J Med. 2004, 350: 2050-2059.

    Article  Google Scholar 

  64. 64.

    Zhou ZG, Hu M, Li Y, Lei WZ, Yu YY, Cheng Z, Li L, Shu Y, Wang TC: Laparoscopic vs open total mesorectal excision with anal sphincter preservation for low rectal cancer. Surg Endosc. 2004, 18: 1211-1215. 10.1007/s00464-003-9170-1.

    Article  PubMed  Google Scholar 

  65. 65.

    Kaiser AM, Kang JC, Chan LS, Vukasin P, Beart RW: Laparoscopic-assisted vs. open colectomy for colon cancer: a prospective randomized trial. J Laparoendosc Adv Surg Tech A. 2004, 14: 329-334.

    Article  PubMed  Google Scholar 

  66. 66.

    Guillou PJ, Quirke P, Thorpe H, Walker J, Jayne DG, Smith AM, Heath RM, Brown JM: Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (MRC CLASICC trial): multicentre, randomised controlled trial. Lancet. 2005, 365: 1718-1726. 10.1016/S0140-6736(05)66545-2.

    Article  PubMed  Google Scholar 

  67. 67.

    Colon Cancer Laparoscopic or Open Resection Study Group: Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomised trial. Lancet Oncol. 2005, 6: 477-484.

    Article  PubMed  Google Scholar 

  68. 68.

    Jayne DG, Brown JM, Thorpe H, Walker J, Quirke P, Guillou PJ: Bladder and sexual function following resection for rectal cancer in a randomized clinical trial of laparoscopic versus open technique. Br J Surg. 2005, 92: 1124-1132. 10.1002/bjs.4989.

    CAS  Article  PubMed  Google Scholar 

  69. 69.

    Braga M, Frasson M, Vignali A, Zuliani W, Civelli V, Di Carlo V: Laparoscopic vs. open colectomy in cancer patients: long-term complications, quality of life, and survival. Dis Colon Rectum. 2005, 48: 2217-2223. 10.1007/s10350-005-0185-7.

    Article  PubMed  Google Scholar 

  70. 70.

    King PM, Blazeby JM, Ewings P, Franks PJ, Longman RJ, Kendrick AH, Kipling RM, Kennedy RH: Randomized clinical trial comparing laparoscopic and open surgery for colorectal cancer within an enhanced recovery programme. Br J Surg. 2006, 93: 300-308. 10.1002/bjs.5216.

    CAS  Article  PubMed  Google Scholar 

  71. 71.

    Arteaga Gonzalez I, Diaz Luis H, Martin Malagon A, Lopez-Tomassetti Fernandez EM, Arranz Duran J, Carrillo Pallares A: A comparative clinical study of short-term results of laparoscopic surgery for rectal cancer during the learning curve. Int J Colorectal Dis. 2006, 21: 590-595. 10.1007/s00384-005-0057-6.

    Article  PubMed  Google Scholar 

  72. 72.

    Liang JT, Huang KC, Lai HS, Lee PH, Jeng YM: Oncologic results of laparoscopic versus conventional open surgery for stage II and III left-sided colon cancers: a randomized controlled trial. Ann Surg Oncol. 2007, 14: 109-117.

    Article  PubMed  Google Scholar 

  73. 73.

    Pechlivanides G, Gouvas N, Tsiaoussis J, Tzortzinis A, Tzardi M, Moutafidis M, Dervenis C, Xynos E: Lymph node clearance after total mesorectal excision for rectal cancer: laparoscopic versus open approach. Dig Dis. 2007, 25: 94-99. 10.1159/000099176.

    Article  PubMed  Google Scholar 

  74. 74.

    Braga M, Frasson M, Vignali A, Zuliani W, Capretti G, Di Carlo V: Laparoscopic resection in rectal cancer patients: outcomes and cost-benefit analysis. Dis Colon Rectum. 2007, 50: 464-471. 10.1007/s10350-006-0798-5.

    Article  PubMed  Google Scholar 

  75. 75.

    Janson M, Lindholm E, Anderberg B, Haglind E: Randomized trial of health-related quality of life after open and laparoscopic surgery for colon cancer. Surg Endosc. 2007, 21: 747-753. 10.1007/s00464-007-9217-9.

    Article  PubMed  Google Scholar 

  76. 76.

    Jayne DG, Guillou PJ, Thorpe H, Quirke P, Copeland J, Smith AMH, Heath RM, Brown JM: Randomized trial of laparoscopic-assisted resection of colorectal carcinoma: 3-year results of the UK MRC CLASICC trial group. J Clin Oncol. 2007, 25: 3061-3068. 10.1200/JCO.2006.09.7758.

    Article  PubMed  Google Scholar 

  77. 77.

    Fleshman J, Sargent DJ, Green E, Anvari M, Stryker SJ, Beart RW, Hellinger M, Flanagan R, Peters W, Nelson H: Laparoscopic colectomy for cancer is not inferior to open surgery based on 5-year data from the COST Study Group Trial. Ann Surg. 2007, 246: 655-664. 10.1097/SLA.0b013e318155a762.

    Article  PubMed  Google Scholar 

  78. 78.

    Chung CC, Ng DCK, Tsang WWC, Tang WL, Yau KKK, Cheung HYS, Wong JCH, Li MKW: Hand-assisted laparoscopic versus open right hemicolectomy: a randomized controlled trial. Ann Surg. 2007, 246: 728-733. 10.1097/SLA.0b013e318123fbdf.

    Article  PubMed  Google Scholar 

  79. 79.

    Ramacciato G, D'Angelo F, Aurello P, Nigri G, Valabrega S, Pezzoli F, Ravaioli M, Cescon M, Cucchetti A, Lauro A, Del Gaudio M, Ercolani G: Right hemicolectomy for colon cancer: a prospective randomised study comparing laparoscopic vs open technique. Chir Ital. 2008, 60: 1-7.

    PubMed  Google Scholar 

  80. 80.

    Lacy AM, Delgado S, Castells A, Prins HA, Arroyo V, Ibarzabal A, Pique JM: The long-term results of a randomized clinical trial of laparoscopy-assisted versus open surgery for colon cancer. Ann Surg. 2008, 248: 1-7. 10.1097/SLA.0b013e31816a9d65.

    Article  PubMed  Google Scholar 

  81. 81.

    King PM, Blazeby JM, Ewings P, Kennedy RH: Detailed evaluation of functional recovery following laparoscopic or open surgery for colorectal cancer with an enhanced recovery programme. Int J Colorectal Dis. 2008, 23: 795-800. 10.1007/s00384-008-0478-0.

    CAS  Article  PubMed  Google Scholar 

  82. 82.

    Ng SSM, Leung KL, Lee JFY, Yiu RYC, Li JCM, Teoh AYB, Leung WW: Laparoscopic-assisted versus open abdominoperineal resection for low rectal cancer: a prospective randomized trial. Ann Surg Oncol. 2008, 15: 2418-2425. 10.1245/s10434-008-9895-0.

    Article  PubMed  Google Scholar 

  83. 83.

    Hewett PJ, Allardyce RA, Bagshaw PF, Frampton CM, Frizelle FA, Rieger NA, Smith JS, Solomon MJ, Stephens JH, Stevenson ARL: Short-term outcomes of the Australasian randomized clinical study comparing laparoscopic and conventional open surgical treatments for colon cancer: the ALCCaS trial. Ann Surg. 2008, 248: 728-738. 10.1097/SLA.0b013e31818b7595.

    Article  PubMed  Google Scholar 

  84. 84.

    Biondi-Zoccai GGL, Lotrionte M, Abbate A, Testa L, Remigi E, Burzotta F, Valgimigli M, Romagnoli E, Crea F, Agostoni P: Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ. 2006, 332: 202-209. 10.1136/bmj.38693.516782.7C.

    Article  PubMed  PubMed Central  Google Scholar 

  85. 85.

    Poolman RW, Abouali JAK, Conter HJ, Bhandari M: Overlapping systematic reviews of anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon-bone autograft: why are they different?. J Bone Joint Surg Am. 2007, 89: 1542-1552. 10.2106/JBJS.F.01292.

    Article  PubMed  Google Scholar 

  86. 86.

    Jadad AR, Cooks DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses. A comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998, 280: 278-80. 10.1001/jama.280.3.278.

    CAS  Article  PubMed  Google Scholar 

  87. 87.

    Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, Barrowman N: When and how to update systematic reviews. Cochrane Database Syst Rev. 2008, 1: MR000023-

    PubMed  Google Scholar 

  88. 88.

    French SD, McDonald S, McKenzie JE, Green SE: Investing in updating: how do conclusions change when Cochrane systematic reviews are updated?. BMC Med Res Methodol. 2005, 5: 33-10.1186/1471-2288-5-33.

    Article  PubMed  PubMed Central  Google Scholar 

  89. 89.

    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010, 340: c365-10.1136/bmj.c365.

    Article  PubMed  Google Scholar 

  90. 90.

    COLOR II Study Group: COLOR II: A randomized clinical trial comparing laparoscopic and open surgery for rectal cancer. Dan Med Bull. 2009, 56: 89-91.

    Google Scholar 

  91. 91.

    American College of Surgeons Oncology Group: Laparoscopic-assisted resection or open resection in treating patients with stage IIA, stage IIIA, or stage IIIB rectal cancer. []

  92. 92.

    Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR: Practical methods for incorporating summary time-to-event data into meta-analysis. Trials. 2007, 8: 16-10.1186/1745-6215-8-16.

    Article  PubMed  PubMed Central  Google Scholar 

  93. 93.

    Williamson PR, Smith CT, Hutton JL, Marson AG: Aggregate data meta-analysis with time-to-event outcomes. Statist Med. 2002, 21: 3337-3351. 10.1002/sim.1303.

    Article  Google Scholar 

  94. 94.

    Parmar MKB, Torri V, Stewart L: Extracting summary statistics to perform meta-analysis of the published literature for survival endpoints. Statist Med. 1998, 17: 2815-2834. 10.1002/(SICI)1097-0258(19981230)17:24<2815::AID-SIM110>3.0.CO;2-8.

    CAS  Article  Google Scholar 

  95. 95.

    Moher D, Liberati A, Tetzlaff J, Altman D, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


This work was supported by an Operating Grant from the Canadian Society of Colon and Rectal Surgeons (GM, DAF, RPB). GM is supported by a Fellowship Award in the Area of Knowledge Translation from the Canadian Institutes of Health Research. The University of Aberdeen Health Services Research Unit is funded by the Scottish Government Health Executive. The views expressed are those of the authors and not necessarily those of the funders (CRR). Podium presentations were made at the Canadian Surgery Forum, Quebec City, Canada, 4 September 2010, and at the Annual Meeting of the Society for Clinical Trials, Vancouver, Canada, 18 May 2011.

Author information



Corresponding author

Correspondence to Guillaume Martel.

Additional information

Competing interests

The Division of General Surgery and Department of Surgery at the Ottawa Hospital, University of Ottawa, are supported by unrestricted educational grants from Covidien Canada and Storz Canada.

Authors' contributions

GM contributed to the conception and design of the study, provided clinical expertise, acquired, abstracted, analyzed, and interpreted the data, and drafted the manuscript. SD acquired, abstracted, analyzed, and interpreted the data. JSB contributed to the conception and design of the study, and revised the manuscript critically. RPB contributed to the conception and design of the study, provided clinical expertise, and revised the manuscript critically. CRR contributed to the conception and design of the study, analyzed and interpreted the data, and revised the manuscript critically. DAF contributed to the conception and design of the study, analyzed and interpreted the data, revised the manuscript critically, and provided global supervision of the project. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Martel, G., Duhaime, S., Barkun, J.S. et al. The quality of research synthesis in surgery: the case of laparoscopic surgery for colorectal cancer. Syst Rev 1, 14 (2012).

Download citation


  • colorectal cancer
  • laparoscopy
  • surgery
  • systematic review
  • umbrella review