What the systematic review of HPV vaccine clinical study reports does, and does not, reveal: commentary on Jørgensen et al.
Systematic Reviews volume 9, Article number: 41 (2020)
Getting hold of, and analysing, clinical study reports (CSRs) for a substantial number of trials of HPV vaccines was a massive undertaking. Nearly 60,000 pages worth of CSRs were amassed, with potentially unique unpublished data points in the tens of thousands. In the end, though, I believe the systematic review by Jørgensen, Gøtzsche, and Jefferson  demonstrates the risks of relying too much on CSRs.
Journal articles are the usual clinical trial reports that systematic reviewers gather and analyse. Articles typically condense a possibly vast amount of data and methodological detail into just a few pages, perhaps with supplementary data files. That is a highly selective process—often misleadingly so. CSRs, on the other hand, do not have space constraints. The longest CSR package included in this systematic review had 11,456 pages, and even the shortest was 357 pages long.
More than 3 years after they began requesting CSRs, Jørgensen and colleagues  were still missing trials for over 21% of all the people who participated in eligible studies. That means there is far more than 21% missing data for some outcomes not fully reported in the available CSRs. That is such a substantial amount, that, as the authors acknowledge, getting hold of it could up-end outcomes at the margins of statistical significance.
When the authors compared meta-analyses drawn from CSR data with those from data drawn from journal articles and/or a clinical trial registry, they found no important differences . Although they argue that theirs is the first study to undertake this type of comparison, there have been similar studies. Results of those I found were mixed [4,5,6,7]. From the little evidence in these, plus the new Jørgensen paper, the absence of CSR data cannot be assumed to render the estimates calculated in a systematic review based on articles unreliable.
The principal value of CSRs seems to lie in data they contain that is not available in any other document. Unfortunately, Jørgensen and colleagues’ methods paper does not include reliable estimates of the extent and importance of data missing from systematic reviews of journal publications, as the authors only compared CSR reports to a single journal article per trial . That means potentially substantial amounts of published data were left out of this comparison. Similarly, the comparison of CSR reports to trial registry information is limited to a single registry, potentially overlooking additional information available on manufacturer websites and other registries.
For example, the 2018 Cochrane review of HPV vaccines included up to seven published articles for a single trial, with additional data from published pooled analyses that included the trial as well . When Jørgensen et al. claim, for example, “If our systematic review of clinical study reports had relied on trial register entries or journal publications, it would have had no data for a quarter of our prespecified outcomes” , their findings do not relate to doing a systematic review based on all journal publications and register entries of a trial.
Turning to the outcomes of the HPV vaccines, Jørgensen and colleagues’ results show broadly similar benefits to those Arbyn et al. found from reviewing trial publications in their recent Cochrane review . Take, for example, the rate of cervical lesions graded CIN 2 or higher (CIN 2+). That is the level of possible cervical cancer precursor that would typically lead to treatment for women in economically advantaged countries. CIN 2+ is used as a measure because it generally takes many years for cervical cancer to develop after HPV infection. That means these trials were not long enough to assess cancer outcomes meaningfully. For perspective, by 1996, CIN 2+ progressed to invasive cancer for about 15% of women in a UK cervical screening programme from 1976 . About a third of women with cervical cancer die within 5 years in the USA .
The Jørgensen review found a risk ratio or relative risk (RR) of 0.81 [0.68–0.97] for CIN 2+ based on 1 to 4 years of follow-up, or about a 20% reduction (from approximately 4.9% to 3.8%) (Additional file 4, analysis 3.7). The Arbyn review found an RR of 0.79 [0.65–0.97] for women who had received at least one dose of vaccine, based on follow-up mostly around 4 years, with one trial over 8 years (analysis 3.7). That was a reduction from approximately 5.1% to 4.0% in a few years.
The most influential conclusions of this systematic review, however, are likely to be the claims about serious and rare neurological harms, based on what the authors make clear are post hoc exploratory analyses. That is extremely worrying because I believe the authors are on very shaky ground here. The conclusions are shaky not just because of the risk of missing data overturning the findings as the authors discuss. To understand how troublesome I find their claims, we need to go back into the early stages of their study.
The original protocol for the systematic review envisaged gaining access to a rich source of detailed data on adverse events from CSRs. In particular, they wanted to be able to assess the risk of two rare neurological conditions, postural orthostatic tachycardia syndrome (POTS), and complex regional pain syndrome (CRPS). They point out in their protocol that both these conditions are exclusionary diagnoses: they could only apply if other, far more likely, causes of symptoms are ruled out.
However, when the yield of data was disappointing, the authors made two amendments to their protocol , developing more indirect ways of trying to assess potential harms as they realised their original plans were not feasible. In my view, this undercuts the methodological rigour of their work.
This, for example, is how they ultimately arrived at something they call “harms judged as ‘definitely associated’ with” POTS or CRPS. They collected every unique term used for any recorded adverse event and put them into an Excel sheet. They asked a single clinician to code those she considered definitely associated with POTS or CRPS. The result, as the authors point out, included conditions “that do not align well with the diagnostic criteria of POTS or CRPS”, like constipation. Coded “definitely associated” was a very long list of symptoms including many kinds of common pain, conditions including food poisoning, and having tests including chest X-rays, blood tests, and ultrasounds. There did not have to be a cluster of them. These events are exceedingly more likely not to be associated with POTS or CRPS than they are to be a signal of a rare neurological condition.
The next methodological issue I found problematic was their conclusion, “we found that the vaccines caused serious neurological harms”. They had data classified as serious neurological events, but they did not know how many separate individuals experienced them. So if a person had a headache bad enough to interfere with their normal activity as well as dizziness that affected them as badly, or they had disturbed sleep (or all three), then that one person would be counted as two (or three) people with serious neurological harms.
The problems inherent in the underlying data were exacerbated by the use of statistical methods that, in my opinion, systematically distorted the presentation of rates for association with POTS and CRPS and serious neurological harms. The authors computed rates that they presented as risk ratios, and, derived from those, numbers needed to harm (NNH). Both of these statistics unambiguously require knowing how many individuals were affected by harms as a proportion of all individuals [12, 13]—data that the authors did not have. You cannot know the risk of being harmed if you do not know how many people were harmed.
The authors drew the line at continuing this method of calculation in cases where events were so common that the numerators eventually exceeded denominators. The results of a meta-analysis in these circumstances, they wrote, would be “nonsensical”. But the respective size of the data points is not what compromises these analyses. The problem is doing calculations with data points other than those the formulas require.
The authors argued that statistically exaggerating the rate of harms would be acceptable because adverse effects are likely to be under-ascertained. For example, any that might theoretically be caused by vaccine adjuvants would be hidden as the comparison groups were not placebos. They also point out that POTS and CRPS could be under-diagnosed. We cannot be sure, though, of the potential magnitude of any of this. What is certain is the rates being used to conclude vaccines cause serious nervous system harm and definite associations with CRPS and POTS are misnamed, and thus, misleading. As the authors report, their analyses show no statistically significant increase in any individual serious or fatal adverse event, or overall serious or fatal harms.
The authors also argue that events classified as serious neurological ones were so uncommon, duplication was unlikely. That is contestable, in my view, given the nature of those events (headaches, to a large extent). If the estimates are accurate, however, and also not a statistical fluke, the risk they calculated is very small: 0.15% versus 0.09%, an absolute difference of 0.06% (or six out of every 10,000).
Against that, we need to weigh the far more common harms caused by cervical cancer, and the surgical procedures used to diagnose it or rule it out. An estimated 2,000,000 women have abnormal Pap smear results in a year in the USA . The National Cancer Institute estimates that 0.6% of women in the USA (or six out of every 1000) will be diagnosed with cervical cancer in their lifetime, and a third of them will die within 5 years. That would be two deaths for every 1000 women in the country . Any substantial reduction in cervical cancer and its potential precursors will prevent anxiety and suffering on a very large scale.
This systematic review confirms that participants in these trials experienced a reduction in possible early signs of HPV-related cancers and the distressing surgical and non-surgical procedures undergone to treat abnormalities. Since the cut-off date for data inclusion in both the Jørgensen and Arbyn systematic reviews, a study following up trial participants for over 10 years has reported a statistically significant drop in cancer, too .
In practice, HPV vaccines are generally used at younger ages than in the trials, when the chance of already being exposed to the viruses is lower. Large-scale vaccination programmes and vaccination of boys might result in some herd immunity, and vaccines that protect against more strains of HPV than those in the trials are in use. Some estimate that the rate of cervical cancer in countries with high vaccination rates could be reduced by half or more in the next few years [8, 16, 17].
Publicity about safety concerns led to substantial drops in HPV vaccination in several countries [18,19,20]. So the stakes in discussing potential vaccine harms are high, both in the need to openly scrutinise the potential for harm and the need to do it responsibly. Only a very rigorous assessment could move us forward. I do not believe the Jørgensen et al. systematic review provides that.
Jørgensen L, Gøtzsche PC, Jefferson T. Benefits and harms of the human papillomavirus (HPV) vaccines: systematic review with meta-analyses of trial data from clinical study reports. Syst Rev. 2020;9(1). https://doi.org/10.1186/s13643-019-0983-y.
Jørgensen L, Gøtzsche PC, Jefferson T. Benefits and harms of the human papillomavirus (HPV) vaccines: comparison of trial data from clinical study reports with corresponding trial register entries and journal publications. Syst Rev. 2020. 2020. Volume 9, Issue 1. https://doi.org/10.1186/s13643-020-01300-1
Jørgensen L, Doshi P, Gøtzsche PC, Jefferson T. Challenges of independent assessment of potential harms of HPV vaccine. BMJ. 2018;362:k3694. https://doi.org/10.1136/bmj.k3694.
Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326:1171–3. https://doi.org/10.1136/bmj.326.7400.1171.
MacLean CH, Morton SC, Ofman JJ, et al. How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epi. 2003;56(1):44–51 https://www.jclinepi.com/article/S0895-4356(02)00520-6/fulltext.
Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737. https://doi.org/10.1136/bmj.c4737.
Rohner E, Grabik M, Tonia T, et al. Does access to clinical study reports from the European Medicines Agency reduce reporting biases? A systematic review and meta-analysis of randomized controlled trials on the effect of erythropoiesis-stimulating agents in cancer patients. PLOS One. 2017;12(12):e0189309. https://doi.org/10.1371/journal.pone.0189309.
Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev. 2018;5:CD009069. https://doi.org/10.1002/14651858.CD009069.pub3.
Raffle AE, Alden B, Quinn M, et al. Outcomes of screening to prevent cancer: analysis of cumulative incidence of cervical abnormality and modelling of cases and deaths prevented. BMJ. 2003;326:901. https://doi.org/10.1136/bmj.326.7395.901.
National Cancer Institute Surveillance. Epidemiology, and End Results Program (SEER). Cancer stat facts: cervical cancer. Bethesda: National Cancer Institute; 2019. Available from: https://seer.cancer.gov/statfacts/html/cervix.html
Jørgensen L, Gøtzsche PC, Jefferson T. Protocol amendment no. 1 and 2: benefits and harms of the human papillomavirus vaccines: systematic review of industry and non-industry study reports. York: PROSPERO; 2017. Available from: https://www.crd.york.ac.uk/PROSPEROFILES/56093_ PROTOCOL_20171116.pdf
Altman D, Machin D, Bryant T, Gardner M. Statistics with confidence: confidence intervals and statistical guidelines. Second edition. London: BMJ Books; 2000.
Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. Version 5.1.0. London: Cochrane Collaboration; 2011.
Insinga RP, Glass AG, Rush BB. Diagnoses and outcomes in cervical cancer screening: a population-based study. Am J Obstet Gynecol. 2004;191(1):105–13. https://doi.org/10.1016/j.ajog.2004.01.043.
Luostarinen T, Apter D, Dillner J, et al. Vaccination protects against invasive HPV-associated cancers. Int J Cancer. 2018;142(10):2186–7. https://doi.org/10.1002/ijc.31231.
Pollock KG, Kavanagh K, Potts A, et al. Reduction of low- and high-grade cervical abnormalities associated with uptake of the HPV bivalent vaccine in Scotland. Br J Cancer. 2014;111(9):1824–30. https://doi.org/10.1038/bjc.2014.479.
Hall MT, Simms KT, Lew J-B, et al. Projected future impact of HPV vaccination and primary HPV screening on cervical cancer rates from 2017–2035: example from Australia. PLOS One. 2018;13(2):e0185332. https://doi.org/10.1371/journal.pone.0185332.
Morimoto A, Ueda Y, Egawa-Takata T, et al. Effect on HPV vaccination in Japan resulting from news report of adverse events and suspension of governmental recommendation for HPV vaccination. Int J Clin Oncol. 2014;29(3):549–55. https://doi.org/10.1007/s10147-014-0723-1.
Corcoran B, Clarke A, Barrett T. Rapid response to HPV vaccination crisis in Ireland. Lancet. 2018;391(10135):P2103. https://doi.org/10.1016/S0140-6736(19)30854-7.
Suppli CH, Hansen ND, Rasmussen M, et al. Decline in HPV-vaccination uptake in Denmark – the association between HPV-related media coverage and HPV-vaccination. BMC Pub Health. 2018;1360. https://doi.org/10.1186/s12889-018-6268-x.
I am grateful to Professor Thomas Lumley, Chair in Biostatistics at The University of Auckland, for statistical advice during the preparation of this commentary, and to Professor Jenny Doust for reviewing it. The views expressed, and any errors, are my own.
I worked at IQWiG between 2004 and 2011, and where I participated in review of data on HPV vaccines as basis for information to the general public, and the development of the Institute’s scientific methods. IQWiG included CSRs in its drug evaluations. I have both collaborated, and been in disagreements, with two of the authors (PCG and TJ). In 2018, I blogged about the authors’ critique of the Cochrane review on HPV vaccines and the dispute between PCG and the Cochrane Collaboration.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Bastian, H. What the systematic review of HPV vaccine clinical study reports does, and does not, reveal: commentary on Jørgensen et al.. Syst Rev 9, 41 (2020). https://doi.org/10.1186/s13643-020-01299-5