Table 1 Quality assessment tool

Data quality

Does the study report information about the research population?

Does the study report what type of data has been used for the analysis, whether it is primary data (collected initially for the study) or secondary data (from a different source)?

If the study used secondary data does the study name, the databases which have been used for the analysis?

If the study has used different databases, does the study describe any linkage between the databases?

Does the study report whether the data set is a representative data set?

Preliminary analyses

Does the study report any statistical procedures used to modify raw data before the analysis?

Primary analyses

Does the study describe about the purpose of the analysis?

Does the study identify the variables used in the analysis and summarize each with descriptive statistics?

Does the study describe the main methods fully, for analyzing the primary objectives of the study?

Is the study clear on which method is used for each analysis, rather than just listing all statistical methods used in one place?

If the method includes any assumptions, does the study verify that the data conformed to the assumptions of the test used to analyze them?

Does the study indicate whether and how any allowance or adjustments were made for multiple comparisons (performing multiple hypothesis tests on the same data)?

Does the study report how it deals with missing data?

If relevant, does the study report how any outlying data were treated in the analysis?

Does the study report the alpha level (e.g., 0.05) that defines statistical significance?

Does the study report the name of the statistical package or program used in the analysis?

If the study needs to control any variables for its objective, does the study report it properly?

Supplementary analyses

Does the study describe sensitivity analyses if applicable?

Does the study test for the underlying assumptions of the methods used in the analysis?

Does the study identify post hoc analyses, including unplanned subgroup analyses, as exploratory?

If there is an imbalance that exists in the outcome variable of the data set, does the study report how the training phase overcome this issue?

Reporting numbers and descriptive stat avg

Does the study report numbers—especially measurements—with an appropriate degree of precision. For ease of comprehension and simplicity, rounded to a reasonable extent?

Does the study report total sample and group sizes for each analysis?

Does the study report numerators and denominators for all percentages?

Does the study summarize data that are approximately normally distributed with means and standard deviations (SD)?

Use the form: mean (SD), not mean ± SD?

Does the study summarize data that are not normally distributed with medians and interpercentile ranges, ranges, or both (report the upper and lower boundaries of interpercentile ranges and the minimum and maximum values of ranges, not just the size of t)?

Does the study report the variability of the data set using either standard deviations, inter-percentile ranges, or ranges (the SE is an inferential statistic—it is about a 68% confidence interval—not a descriptive statistic)?

Does the study display summarized or exact data in tables?

Does the study display data in figures? Tables present exact values, and figures provide an overall assessment of the data?

Reporting risk and ratios

Does the study describe the type of rate (e.g., incidence rates; survival rates), ratio (e.g., odds ratios; hazard ratios), or risk (e.g., absolute risks; relative risk differences), being reported?

Does the study describe the quantities represented in the numerator and denominator?

Does the study report the time period over with each rate applies?

Does the study report any unit of population (that is, the unit multiplier: e.g., × 100; × 10,000) associated with the rate?

Does the study consider reporting a measure of precision (a confidence interval) for estimated risks, rates, and ratios?

Validation

Does the study describe methods of validation used in the training phase (e.g., cross validation, use of test/hold-out sample)?

Does the study describe the attempts to generalize the model beyond the immediate context?

Regression analysis

Does the study describe the purpose of the analysis?

Does the study confirm that the assumptions of the analysis were met? For example, in linear regression indicate whether an analysis of residuals confirmed the assumptions of linearity.

Does the study report the regression equation for either simple or multiple (multivariable) regression analyses?

For primary comparisons analyzed with simple linear regression analysis, does the study consider reporting the results graphically, in a scatter plot showing the regression line and its confidence bounds?

Does the study report the alpha level used in the univariate analysis?

Does the study report whether the variables were assessed for collinearity?

Does the study report whether variables were assessed for interactions?

Does the study describe the variable selection process by which the final model was developed (e.g., forward stepwise; best subset).

Does the study report the regression coefficients (beta weights) of each explanatory variable and the associated confidence intervals and P values, preferably in a table?

Does the study provide a measure of the model’s “goodness-of-fit” to the data (the coefficient of determination, r2, for simple regression and the coefficient of multiple determination, R2, for multiple regression)?

Survival analysis

Does the study describe the purpose of the analysis?

Does the study describe the dates or events that mark the beginning and the end of the time period analyzed?

Does the study specify the circumstances under which data were censored?

Does the study specify the statistical methods used to estimate the survival rate?

Does the study confirm that the assumptions of survival analysis were met?

For each group, give the estimated survival probability at appropriate follow-up times, with confidence intervals, and the number of participants at risk for death at each time. It is often more helpful to plot the cumulative probability of not surviving.

For each group, give the estimated survival probability at appropriate follow-up times, with confidence intervals, and the number of participants at risk for death at each time. It is often more helpful to plot the cumulative probability of not surviving.

Reporting median survival times, with confidence intervals, is often useful to allow the results to be compared with those of other studies?

Does the study present the full results in a graph (e.g., a Kaplan-Meier plot) or table?

Does the study specify the statistical methods used to compare two or more survival curves?

Does the study report the P value, when comparing two or more survival curves with hypothesis tests?

Does the study report the regression model used to assess the associations between the explanatory variables and survival or time-to-event?

Does the study report a measure of risk (e.g., a hazard ratio) for each explanatory variable, with a confidence interval?

SEM models

Does the study report all the parameters and their standard errors?

Does the study report the reason for the choice of a clear and complete form of path model structure?

Does the study report the global indices of fit?

Does the study provide reasons as justification for omitted directed and non-directed arcs?

Does the study report alternative and equivalent models?