Our findings suggest that key reporting items are missing in the majority of published NMAs. While minor improvements in the reporting of some elements were noted following PRISMA-NMA, other elements also experienced reductions. In total, reporting has improved after 2015 in 47% of the modified PRISMA-NMA items, but it has also deteriorated in 49% of the core items. Improvement was observed in items about the NMA synthesis, overview of the network and study characteristics, while deterioration was noticed in the description of summary effect sizes to be used, presentation of individual study data, sources of funding for the systematic review, and role of funders.
An explanation of the decrease in inadequately reported items may be restriction in the word count required by most journals. Also, some of the required details may be reported in the study’s protocol. Another key item that was inadequately reported was funding status. Journal guidelines highlight funding as crucial information to be reported in a paper; however, we noticed that mainly the author funding is reported and not funding for the review itself. Funding for the systematic review and role of funders are consistently underreported, which is a cause of concern. Presentation of individual study data is another item that has worsened after 2015. This may be because larger and more complex networks are being structured compared to past years or because of authors’ desire to retain ownership of the data, given the large efforts to compile the data sets, and to potentially publish new work after additional evidence (e.g. treatments) emerge.
Key factors that may impact the reporting were the journal’s impact factor, funding type, year of publication, type of review, and treatment category included in the network. In particular, newer and publicly sponsored NMAs of non-pharmacological therapies with a protocol, and published in high impact factor journals, were associated with better reporting. Our results showed that reviews with a protocol, and particularly Cochrane reviews, were associated with higher PRISMA-NMA scores.
Overall, reporting is adequate but not high (mean PRISMA-NMA score 32.1; 95% CI 31.8–32.4; max 49). Authors of NMAs showed a steep improvement in earlier years (2013–2015), but it stabilises after the PRISMA-NMA guideline publication. The improvement continues to exist throughout the years, but the speed of improvement is lower between 2016 and 2018. This suggests that overall, the PRISMA-NMA guidance has not importantly affected reporting in new NMAs compared to older NMAs. This may be because NMA authors in the earlier years 2013–2015 already followed existing guidelines for standards of conduct of NMA through the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) tools [16, 17] and National Institute for Clinical Excellence (NICE) Decision Support Unit’s Evidence Synthesis Technical Support Documents (TSDs) documents [18]. However, we observed improvement in the five items (S1–S5) specific to the reporting of the synthesis section of an NMA after 2015, ranging between 4 and 12%. The improvement observed in reporting might be attributed at least in part to PRISMA-NMA, but may also be due to additional factors, such as the increase in registering or publishing of peer-reviewed protocols; protocol existence in NMAs has increased from 15–39% between 2013 and 2018. Our analysis showed that there is a slight improvement in reporting in the year 2016 compared to the year 2017 (2016: mean modified PRISMA-NMA score 33.0, 95% CI 32.3–33.8; 2017: mean modified PRISMA-NMA score 31.2, 95% CI 30.6–31.8). This may be due to the impact factor of the journal that NMAs were published in. The median impact factor of the journals the NMAs were published in 2016 was 3.87 (IQR 2.49–5.56), whereas in 2017 was 3.50 (IQR 2.63–5.16) (Appendix Fig. 8).
In our database of NMAs, the PRISMA-NMA guideline is only endorsed by 7% of the journals in which the NMAs were published. This highlights the need for journals publishing systematic reviews and NMAs to adopt the PRISMA-NMA guidelines to improve reporting, and to request the checklist upon a manuscript submission. Based on our findings, we provide recommendations to update the PRISMA-NMA statement to facilitate its use by systematic reviewers, journal editors, and peer reviewers (see the section “Conclusions and recommendations for practice”).
To the best of our knowledge, this is the largest review assessing the PRISMA extension guideline for NMA in more than 1000 systematic reviews and NMAs. Our findings are aligned with previous findings by Hutton et al. [11], who evaluated 89 NMAs of non-pharmacological therapies; Tonin et al. [10] who assessed 477 NMAs of pharmacological treatments; and Lee and Shin [9] who assessed 21 NMAs in dental care. In agreement with assessments in systematic reviews and meta-analyses, reporting in the 27 core PRISMA items was suboptimal [2]. A previous assessment on reporting of pairwise systematic reviews and meta-analyses on nursing interventions in patients with Alzheimer’s disease before and after PRISMA publication showed an improvement in the average core PRISMA items from 17.11 to 20.83 score [19]. Our findings about PRISMA-NMA are not limited to a specific disease area and showed that the average core PRISMA items score did not importantly change before (19.58 items) and after 2015 (19.88 items).
In addition to assessing the PRISMA-NMA items in the included systematic reviews, we explored factors that play a key role in reporting of NMA. In agreement with Zarin et al. [7], we found that the prerequisite assumptions are not always considered; 28% of NMAs (265 of 958 NMAs with a closed loop) did not report an assessment for consistency in their methods. However, similar to Petropoulou et al.’s [8] findings, reporting improved a bit over the years.
A limitation of our study is that in our assessment we considered that a PRISMA component was reported only if relevant information was present in the underlying section of the manuscript, as indicated in the PRISMA-NMA guideline. Also, protocols were not assessed for reporting relevant details, since the PRISMA-NMA guideline refers only to the final manuscript for NMAs. In our study, we have not explored differences in NMAs pointing authors to the relevant protocols for methods details and the remaining NMAs. However, in our assessment, we considered all available supplementary files and appendices. Although we may have missed some details reported in the protocol, we expect that this could not importantly impact our results. Also, an important unmeasured confounder in reporting may have been journals with no word count restrictions, but we have not assessed this further. Another potential limitation is that our literature search was conducted up to July 2018, and we may have missed recently published NMAs that were reported well. Also, the impact of the PRISMA-NMA guideline may not immediately be seen in the reporting of published NMAs and may take more time to start using it. However, this is the largest NMA database that assessed reporting, and we expect that no major differences would be seen in our results regarding the overall trend in reporting. A risk of confounding may be associated with our results between industry-sponsored studies and pharmacological treatments. We found that both factors were associated with a decrease in reporting (of the 170 industry-sponsored NMAs, 162 [95%] included pharmacological interventions only [with or without a placebo] in the network). We used a binary system (presence/absence) for the PRISMA-NMA items, but this may not be the best approach to assess adequacy of reporting. For example, authors may report that transitivity was assessed but without providing more details on this.
Conclusions and recommendations for practice
NMAs published after 2015 more frequently reported the five items associated with NMA (i.e. description of methods to explore network geometry, description of methods to assess inconsistency, network plot presentation, brief overview of network characteristics, description of results from investigations of inconsistency). However, several important items are underreported and the yearly improvement in reporting is small.
In conducting this research, we chose to split certain PRISMA-NMA items into more specific items for evaluation, moving from 32 to 49 items. This allowed us to highlight crucial aspects of NMA that were or were not reported across the years. To this end, we suggest that the PRISMA-NMA checklist be updated using the 49 items instead of the initially suggested 32 items. This will provide more in-depth guidance to review authors, reviewers, editors, and readers for adequate reporting in NMA. The 49 items are listed in Appendix Table 1. Clarifying the information presented in the PRISMA-NMA 32 items into 49 different items may increase word count, but will enhance transparency of reporting. Online appendix files can also be used for additional and supporting information of the systematic review and NMA.
The original or modified PRISMA-NMA guidelines should be used extensively by review authors and be adopted by a wider range of journals. Journals editors, peer-reviewers, and systematic review authors should use the PRISMA-NMA list on a regular basis to write evaluate and publish results from NMA, paying special attention to items that are still underreported as highlighted in Fig. 3.