Protocol for a systematic review of the effects of interventions for vaccine stock management
Systematic Reviews volume 8, Article number: 14 (2019)
Inadequate vaccine stock management in health facilities leads to vaccine stock-outs. The latter threatens the success of immunisation programmes. Countries have used various approaches to reduce stock-outs and improve vaccine availability, but we are not aware of a systematic review of these interventions. This protocol describes the methods we will use to assess the effects of existing approaches for improving vaccine stock management.
We include randomised and non-randomised studies identified through a compehensive search of peer-reviewed and grey literature databases. We will search PubMed, Cochrane Central Register of Controlled Trials, Embase, Web of Science, PDQ-Evidence and Scopus. We will also search websites of the World Health Organisation (WHO), Global Alliance for Vaccine and Immunisation, PATH Vaccine Resources Library and United Nations Children’s Fund. In addition, we will search the WHO International Clinical Trials Registry Platform and reference lists of included studies and relevant reviews. Finally, we plan to do a citation search for included studies. We will use Cochrane recommended methods to screen search outputs, assess study eligibility and risk of bias, extract and analyse study results. We will use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool to assess the certainty of the evidence on the effects of the interventions.
We believe that the findings of this review will serve as valuable information for policy makers on ways to improve vaccine stock management and vaccine availability. When vaccine availability is improved, those who need them, especially children, will be adequately protected from vaccine-preventable diseases.
Systematic review registration
The success of immunisation programmes depends on a well-functioning supply chain that ensures the constant availability of quality vaccines to the target population [1,2,3]. Effective vaccine stock management is one of the criteria for an effective vaccine supplychain [1, 4]. Vaccine stock management at health facility level involves the checking and monitoring of vaccines on arrival at a storage point, during storage and when they are administered to the users [1, 2]. Adequate vaccine stock management helps to maintain the quality of vaccines [1, 2] and prevent vaccine stock-outs. Vaccine stock-outs refer to the absence of vaccine(s) at the point of service delivery to the patient [3, 5]. An analysis of global data on effective vaccine management assessments between 2009 and 2014 showed that most low- and middle-income countries performed below the minimum standard for adequate vaccine stock management .
Recent data reported by countries in World Health Organisation (WHO) and United Nations Children’s Fund (UNICEF) joint reporting show that each year at least one-third of countries experience one or more vaccine stock-outs lasting for at least 1 month . The most vulnerable groups who suffer the effects of vaccine stock-outs in resource-constrained settings are the urban poor and rural communities who depend on public facilities for health services. When vaccines are not available, these recipients of public health services are obliged to make repeated and costly trips to health facilities. Ultimately, immunisation targets are not met, universal health coverage remains an elusive dream and lives are lost .
Due to the upward trend in the rates of vaccine stock-outs, countries are currently creating approaches to improve vaccine stock management [7, 9]. The approaches for improving vaccine availability may include the use of digital systems to monitor vaccine stock levels in real time [10,11,12] . These dashboards measure performance and make them visible for managers to make informed decisions . Another vaccine stock management approach involves the crowd sourcing of reports of stock-outs from patients and community volunteers. These reports are then sent to relevant health system structures to elicit system changes for improving vaccine availability . However, we are not aware of a systematic review of these and other potential interventions for improving vaccine stock management.
We aim to assess the effects of approaches used for vaccine stock management at facility level.
This systematic review protocol has been prepared according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P) 2015 guideline (Additional file 1).
Registration of the review
We registered the systematic review in the International Prospective Register of Systematic Reviews (PROSPERO) .
Eligibility criteria for studies
We will include individually randomised trials, cluster randomised trials, controlled before-after studies, interrupted time series studies and repeated cross-sectional studies. Eligible participants include the healthcare systems which deliver vaccines, healthcare facilities where vaccines are administered, healthcare workers involved in providing immunisation services and recipients of immunisation services.
We plan to include interventions targeting recipients or providers of immunisation services. Recipient-oriented interventions may include the involvement of end-users in monitoring vaccine availability at facilities, e.g. using mobile phone services or hotline platforms. Examples of interventions directed at providers of immunisation services include education or training, audit and feedback, prompts or reminders and supportive supervision. We will also include interventions targeting the health system offering immunisation services, e.g. action plans, re-designing (components) of the supply chain and integration with other services. Other interventions intended to ensure vaccine availability, including multi-component interventions, are also eligible for inclusion. We will consider the following as eligible comparisons: standard vaccine stock management practices in the study setting, alternative interventions and similar interventions implemented with different degrees of intensity.
Our primary outcomes are vaccine availability and vaccine stock-outs. We will measure vaccine availability as the proportion of vaccination days in which vaccines were available and no one eligible for vaccination was turned back for lack of vaccines; but will also consider other measures of vaccine availability used by the authors of included studies. Vaccine stock-out rates in the review will be measured as the percentage of facilities that experienced a stock-out of a specific vaccine that the site is expected to provide, at any point, within a defined period; or other definitions as used in included studies. Our secondary outcomes include acceptability, adverse events and cost of the intervention, as well as other outcomes as reported by included studies.
We will develop a comprehensive search strategy for both peer-reviewed and grey literature. We will search the following databases PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), Embase, WHO Library Information System (WHOLIS), Web of Science, PDQ (Pretty Darn Quick)-Evidence and Scopus. We will also search the websites of WHO, Global Alliance for Vaccine and Immunisation, PATH Vaccine Resources Library and UNICEF. In addition, we will search the WHO International Clinical Trials Registry Platform, reference lists of included studies and related systematic reviews and citations of included studies. A preliminary search strategy developed for PubMed is found in the Appendix.
Data collection and analyses
Two authors will independently screen the titles and summaries of records retrieved from the search for potentially eligible studies. We will obtain full-texts for all the potentially eligible studies. Two authors will assess these full-text publications for eligibility. Any disagreements between the two authors regarding study eligibility will be resolved by discussion and consensus. A third author will arbitrate any unresolved disagreements. We will provide a table with the characteristics of the included studies, and another of excluded studies with reasons for their exclusion. We will seek additional information, for studies with missing information, to assist us in our decision-making process.
For each included study, two authors will independently extract information using a piloted data extraction form. Extracted data will include study design, participant, intervention and outcome characteristics as well as outcome data. Any differences will be resolved through discussion and consensus. A third author will be consulted to arbitrate if disagreements persist between the two authors. If there are missing data, we will contact study investigators to obtain the missing information. Two authors will independently assess risk of bias in included studies using the appropriate tool for randomised trials  and non-randomised studies . Differences in judgement will be resolved by discussion and consensus, with arbitration by a third author.
We will present study results as risk ratios for dichotomous data (e.g. frequency of vaccine stock-outs), and mean differences for continuous data (e.g. duration of vaccine stock-outs) will be presented as mean difference. We will combine data from clinically homogenous studies (in terms of designs, participants, interventions and outcomes) using random-effects meta-analysis. However, if we come across variation between studies, the findings will be summarised in a narrative format. We will analyse results of interrupted time series studies using regression analysis with time trends before and after the interventions [18, 19]. We will assess certainty of the evidence of effects of interventions using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach .
We will look out for and correct any errors made in the analysis of included studies. For example, if clustering is not addressed in an eligible cluster randomised trial, we will re-analyse the data if sufficient information is available. Otherwise, we will request necessary data from the authors or attempt to adjust data for clustering by inflating the standard errors by multiplying them by the square root of the design effect . The adjusted effect will then be added in the meta-analysis. We will assess statistical heterogeneity among study results using the I2 statistic. We will consider heterogeneity as substantial, if the I2 is 50% or more. We will investigate the causes of substantial statistical heterogeneity using subgroup analyses. We will define subgroups based on participant and study design characteristics. We will use the chi-squared test for subgroup differences to assess for subgroup interactions. We will carry out sensitivity analyses, if applicable on aspects that could potentially affect the meta-analysis results such as study designs and overall risk of bias. We may also conduct a sensitivity analysis to explore the effects of fixed- versus random-effects analyses for outcomes with statistical heterogeneity .
This systematic review will examine the effectiveness of existing approaches for managing vaccine stock levels at health facilities, in order to prevent vaccine stock-outs. Study findings will serve as valuable information for policy makers on ways to improve vaccine stock management and vaccine availability. When vaccine availability is improved, target populations will be adequately protected from vaccine-preventable diseases. Furthermore, there will be reduction in the number repeated visits that patients should make in a bid to get vaccinated. This will increase their trust in the health systems. Ultimately, there will be a reduction in deaths caused by vaccine-preventable diseases as well as an improvement in other health outcomes.
Cochrane Central Register of Controlled Trials
Grading of Recommendations Assessment, Development and Evaluation
Global Vaccine Action Plan
International Clinical Trials Registry Platform
United Nations Children’s Fund
World Health Organisation
Asfar A, Kartlogu U. Vaccine stock management: guidelines on stock records for immunization programme and vaccine store managers; 2006.
World Health Organization (WHO). Immunization supply chain and Logistics. A neglected but essential system for national immunization programmes. WHO/IVB/14.05. 2014. https://www.who.int/immunization/documents/WHO_IVB_14.05/en/. Accessed 14 May 2018.
SAGE. Pre-empting and responding to vaccine supply shortages. SAGE April 2016 Executive summary. 2016. http://www.who.int/immunization/sage/meetings/2016/april/1_Mariat_shortages_SAGE_2016.pdf. Accessed 22 Feb 2018.
World Health Organization (WHO). Assessment Report of the Global Vaccine Action Plan. Strategic Advisory Group of Experts on Immunization (SAGE). 2014. http://www.who.int/immunization/global_vaccine_action_plan/SAGE_DoV_GVAP_Assessment_report_2014_English.pdf?ua=1. Accessed 19 Feb 2018.
World health organisation (WHO). Meeting report: technical definitions of shortages and stockouts of medicines and vaccines. World Heal Organ. 2016;http://apps.who.int/medicinedocs/documents/s23146en/s23146en.pdf. Accessed 22 Feb 2018.
WHO/UNICEF. Effective vaccine management( EVM) global data analysis. 2015.
Lydon P, Schreiber B, Gasca A, Dumolard L, Urfer D, Senouci K. Vaccine stockouts around the world: are essential vaccines always available when needed ? Vaccine. 2017;35:2121–6.
Médecins Sans Frontières (MSF), Southern African HIV Clinicians Society. Stop Stockouts Project (SSP) Stockouts National Survey. 2016.
Strategic Advisory Group of Experts on Immunisation (SAGE). The 2017 assessment report of the Global Vaccine Action Plan. Geneva: WHO; 2017.
Shieshia M, Noel M, Andersson S, Felling B, Alva S, Agarwal S, et al. Strengthening community health supply chain performance through an integrated approach: using mHealth technology and multilevel teams in Malawi. J Glob Health. 2014;4:020406.
Kapuria B, Talukdar J, Muthusamy N, Gera R. Designing and implementing an intelligent vaccine logistics management system for India’s Universal Immunisation Programme (UIP) - ‘The eVIN Model’. J Pharm Policy Pract. 2014;7(Suppl 1):O3.
Gilbert SS, Thakare N, Ramanujapuram A, Akkihal A. Assessing stability and performance of a digitally enabled supply chain: retrospective of a pilot in Uttar Pradesh, India. Vaccine. 2017;35:2203–8.
GAVI, WHO, UNICEF, Melina B and Gates foundation. Guidance on Dashboards for Immunization Supply Chains. 2015. https://www.technet-21.org/iscstrengthening/uploads/guidance-on-dashboards.pdf.
Mckenzie A, Mccann T. Evaluation of the stop stock outs project (SSP). South Africa: Stockholm Evaluation Unit; 2016.
Chinwe J, Jaca A, Abdullahi LH, Ngcobo NJ, Wiysonge CS. A systematic review of interventions for vaccine stock management: PROSPERO; 2018. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42018092215. Accessed 14 Apr 2018
Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions; 2011.
Sterne J, Hernán M, Reeves B, Savović J, Berkman N, Viswanathan M, et al. Risk of bias in non-randomized studies of interventions ( ROBINS-I ): detailed guidance. BMJ. 2016;355(October):i4919.
Ramsay CR, Grimshaw JM, Thomas RE. Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. Int J Technol Assess Health Care. 2003;19:613–23.
Abdullahi LH, Kagina BMN, Wiysonge CS, Hussey GD. Improving vaccination uptake among adolescents. Cochrane Database Syst Rev. 2015;(9):CD011895.
The authors would like to thank Elizabeth Pienaar for helping to design the search strategy for this review.
All protocol amendments will be described, dated and justified in the published systematic review and in the registration in PROSPERO.
This work is sponsored by the South African Medical Research Council and the National Research Foundation of South Africa (Grant Numbers: 106035 and 108571).
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Iwu, C.J., Jaca, A., Abdullahi, L.H. et al. Protocol for a systematic review of the effects of interventions for vaccine stock management. Syst Rev 8, 14 (2019). https://doi.org/10.1186/s13643-018-0922-3