Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review

Background Adequately conducted systematic reviews with meta-analyses are considered the highest level of evidence and thus directly defines many clinical guidelines. However, the risks of type I and II errors in meta-analyses are substantial. Trial Sequential Analysis is a method for controlling these risks. Erroneous use of the method might lead to research waste or misleading conclusions. Methods The current protocol describes a systematic review aimed to identify common and major mistakes and errors in the use of Trial Sequential Analysis by evaluating published systematic reviews and meta-analyses that include this method. We plan to include all studies using Trial Sequential Analysis published from January 2018 to January 2022, an estimated 400 to 600 publications. We will search Medical Literature Analysis and Retrieval System Online and the Cochrane Database of Systematic Reviews, including studies with all types of participants, interventions, and outcomes. Two independent reviewers will screen titles and abstracts, include relevant full text articles, extract data from the studies into a predefined checklist, and evaluate the methodological quality of the study using the AMSTAR 2, assessing the methodological quality of the systematic reviews. Discussion This protocol follows the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P). The identified mistakes and errors will be published in peer reviewed articles and form the basis of a reviewed guideline for the use of Trial Sequential Analysis. Appropriately controlling for type I and II errors might reduce research waste and improve quality and precision of the evidence that clinical guidelines are based upon. Supplementary Information The online version contains supplementary material available at 10.1186/s13643-022-01987-4.


PRISMA-P 2015 Checklist
This checklist has been adapted for use with protocol submissions to Systematic Reviews from Table 3  Objectives 7 Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO) 3 Section/topic # Checklist item

Information reported Line number(s) Yes No
language, publication status) to be used as criteria for eligibility for the review

Information sources 9
Describe all intended information sources (e.g., electronic databases, contact with study authors, trial registers, or other grey literature sources) with planned dates of coverage Page 7

Search strategy 10
Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated Supplemental material

Data management 11a
Describe the mechanism(s) that will be used to manage records and data throughout the review Page 7 Selection process 11b State the process that will be used for selecting studies (e.g., two independent reviewers) through each phase of the review (i.e., screening, eligibility, and inclusion in metaanalysis) Page 7 Data collection process 11c Describe planned method of extracting data from reports (e.g., piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators Page 8

Data items 12
List and define all variables for which data will be sought (e.g., PICO items, funding sources), any pre-planned data assumptions and simplifications Page 8

Outcomes and prioritization 13
List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale NA

Risk of bias in individual studies 14
Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis

Record ID __________________________________
Standard procedure for data extraction General workflow Source files via Covidence: Major mistakes and Errors conducted using Trial Sequential Analysis in systematic reviews or meta-analyses (METSA) (covidence.org) Filter by #CovidenceID Add missing files to covidence (e.g. protocol or supplementary material) Go through REDcap instruments 1-5 in the pre-specified order Handling missing data or incompatibility When a numerical value is required but none is available, input: "-99" When text is required but none is available, input: "NA" If question is missing appropriate answer, consult JBM Misc. When citing paper, place citation in " " Records are added using the attached list If in doubt about any of the above, consult CGR or JBM Systematic review (with pre-published/pre-registred protocol) Meta-analysis (without pre-published/pre-registred protocol) (Systematic reviews aim to minimize bias through the use of pre-specified research questions and methods that are documented in protocols)

Was the review an update of a previous systematic Yes review? No
When the review is an update of an existing review, please identify the protocol for updated review. If no protocol has been made specifically for the updated review, assess whether the updated review explicitly and firmly claims to adhere to methods defined in previous protocols/review versions. Otherwise, mark the updated review as having no predefined protocol. How were the subgroup analyses labelled by the "Subgroup analysis" authors? "Secondary" "Exploratory" "Additional" "Observational" Not labelled Other Please describe how the subgroup authors were labelled by the authors __________________________________________ To which outcome(s) and or subgroup analyses was TSA applied? __________________________________________ (Include also outcomes to which TSA was planned, but data was missing or sample size was too low to Authors may choose to apply TSA but not share the perform the TSA) graph and only report TSA CI's -look closely! If applicable, please list outcomes to which TSA was planned, but could not be performed due to lack of data __________________________________________ How many RCT's were included in the largest meta analysis? __________________________________ (Largest by sample size. If the RCT has multiple groups and, therefore, is included in the meta-analysis more than once, count each data entry and make a note in the 'general comment field' below.) How many non-randomized trials were included in the article? __________________________________ How many non-randomized trials were included in the largest meta analysis? __________________________________ (Largest by sample size) Was the meta analysis presented in a forest plot? Yes No The following questions ask for the FINAL conclusions of the study and not the "subconclusions" of the individual analyses, i.e., the results of individual TSA analyses or forest plots What did the authors conclude about the intervention Inconclusive effect on the outcome(s) analyzed with TSA?
Beneficial Harmful Outcome(s) analyzed with TSA: Other/multiple (Reminder to read the entire conclusion -authors may conclude that data indicates effect, only to conclude that the data quality was too low and thus the final conclusion is "inconclusive") Please describe the conclusion (or copy/paste) Which analysis model was used for the outcome analyzed Fixed-effect model with TSA?

__________________________________________
Random-effects models Fixed-and random-effects models Other None Please provide additional model details and/or describe other models __________________________________________

TSA -Methods
This instrument asks for the details of the TSA analysis. Often, only one alpha-level will be chosen for all analyses (TSA, forest plots, etc.) but in the case that alpha-levels (or other details) differ, please note the value used for the TSA analysis and describe the difference in comments at the end of the instrument.
Values for each TSA should be presented in the figure, above the figure (figure head) or below the figure (legend), but will often be found in the description of statistical methods and/or results section. Was heterogeneity correction used for Trial Sequential Inconsistency (I-square) Analysis?
Diversity (D-square) Not described, but no heterogeneity found in meta analysis ( I^2=0% ) Was the AIS directly extracted from figure or figure Directly extracted (e.g., from figure, figure legend or was it calculated by the extractor, e.g., legend or text) from forest plots?
Calculated (e.g., summing sample sizes from forest plot) Did the authors present a required information size Yes, diversity-adjusted required information size for the analysis? Please note in the table below were the individual trials of the Z-curve was placed in the TSA graphic (see area description in image above) -if TSA contains more than 84 studies, please leave a comment using the comment function and contact JBM Trial number TSA graph area (input integer: 1-6) Trial number TSA graph area (input integer: 1-6) Trial number TSA graph area (input integer: Discrepancies between graph and text or between text segments Graphic quality of figure -including labels Specification of included trials Acquired information size Required information size (RIS or DARIS) Relative risk reduction or MIREDIF Heterogeneity Proportion of events in the control population (only dichotomous outcomes) Variance (only continuous outcomes) Chosen association measure Which parameters caused you to rate the overall Discrepancies between graph and text or between transparency as "[tsares_trans]" ?
text segments Graphic quality of figure -including labels Specification of included trials Acquired information size Required information size (RIS or DARIS) Relative risk reduction or MIREDIF Heterogeneity Proportion of events in the control population (only dichotomous outcomes) Variance (only continuous outcomes) Chosen association measure Other Please describe which other parameters influenced your rating __________________________________________