In our overview of systematic reviews examining the effectiveness of reminders in improving professional behavior, we identified 35 systematic reviews with AMSTAR scores ranging from 1 to 8 (out of total score of 11) with a median of 3. The results of the reviews indicate positive effects when reminders are incorporated into a variety of clinical settings for different types of diseases. Furthermore, the results support that modest improvements can occur with the use of reminders, with one review estimating an overall effect size of 4.2%. This effect size is consistent with other interventions that have been used to improve professional behavior
There are several strengths of this overview. First, it employed a comprehensive search strategy, developed and implemented by an information specialist as part of a larger project to examine interventions to change professional behavior. Second, duplicate screening, data extraction and quality assessments were conducted. Third, a validated instrument (AMSTAR) was used to assess the methodological quality of included reviews
There are also several limitations to this overview. First, we did not retrieve data from the primary studies; therefore, we were limited by the information reported by the review authors on aspects such as the description of the interventions and outcomes. However, by focusing on the results of the systematic reviews rather than each individual primary study, we were able to obtain a broad sense of the field. We also focused our conclusions on reviews with AMSTAR scores >5 to address concerns with the quality of the systematic reviews. Second, this overview could not examine differences in effectiveness that may exist between locally developed and commercially available reminder systems due to the limited data. Only three of the included reviews evaluated the effectiveness of locally developed versus commercially available reminder systems
[17, 23, 27]. The subgroup analysis conducted by Ammenwerth and colleagues demonstrated a higher relative risk reduction for locally developed systems, which they suggested was likely because they are developed to meet local needs, and sites often receive additional resources and support when implementing these systems
. Garg and colleagues also found that authors who created the decision support system were more likely to report improved performance
, but this was not supported by Shojania et al.
There may be other factors that impact the effectiveness of reminders, such as the functionality of the decision support system. Functionality (how well a system can model the clinician decision making process) was examined by two of the included reviews
[20, 24]. Both found that systems that actively engaged clinicians (either prompted them to use the system or required a response) improved performance to a greater degree compared with systems that required clinicians to initiate use.
Third, the 35 included systematic reviews are not independent given the significant overlap of the included studies. In total, 655 studies from the 35 reviews were included for analyses in this overview. One hundred and ninety-five studies were analyzed only once (found in only one included review) and 459 were “double-counted.” This overlap occurred most often with larger reviews that included more than 20 primary studies. In fact, 35% (160) of these studies were “double-counted” in the three largest systematic reviews in this overview
[27, 28, 43]. Therefore, we would argue that these reviews should be considered as (partial) replications of systematic reviews of reminders undertaken by different authors with different inclusion criteria and methods. The convergence of findings across the reviews, therefore, is not surprising but reassuring that the findings are not due to the specific inclusion criteria or methods adopted by a group of authors.
Finally, there remains a lack of information on the long-term effect of reminders. None of the reviews restricted inclusion of studies based on length of follow-up, but the majority of studies were of relatively short durations. Whether the effectiveness of reminders diminishes over time has not been established to date.
An important methodological challenge in conducting overviews is how to interpret and summarize an area of research that includes reviews of variable quality. Another challenge is that inevitably, individual studies will be included in more than one systematic review, which leads to “double-counting.” To overcome both of these challenges, in this overview, we included all eligible reviews regardless of quality but we focused our conclusions on those of higher quality (AMSTAR >5). Therefore, we were able to provide a comprehensive picture of the state of the research literature on the effectiveness of reminders.
With 35 reviews included in this overview, and others still continuing to be published, evaluating the effectiveness of reminder interventions is a topic of frequent publication
[51–54]. However, the literature is disorganized and reviews are often published in overlapping topic areas, which suggests that there is an unnecessary duplication of efforts by review authors
. Furthermore, the publication of reviews that focus on specific populations, settings or diseases (that is, split reviews) rather than broadly based reviews that include all professionals in all settings (that is, lumped reviews) also adds to the duplication of efforts since the former are really subgroup analyses of the latter. It is unclear whether further systematic reviews of reminders will likely change the conclusions of this overview, although we judge this unlikely, questioning the need for substantial new reviews in this area. Instead, we would argue that the field would be best served by updating (a limited number of) high quality broad reviews. Future reviews should focus on possible effect modifiers and moderators to explain the variation observed across primary studies of reminders. Given the poor quality of existing reviews, authors of future reviews must utilize more robust methods to conduct and report their reviews. This should lead to fewer but higher quality reviews, resulting in a more organized field of literature that is more interpretable by end-users.