Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews

Table 3 Model results for each dataset while varying all model characteristics

Datasets	Sensitivity mean (SD)	Specificity mean (SD)	Precision mean (SD)	Accuracy mean (SD)	Correct reason for exclusion^a mean (SD)
Psoriasis	84.97% (14.99%)	75.72% (15.84%)	19.82% (9.55%)	76.17% (14.63%)	88.66% (7.76%)
Lung cancer	77.01% (21.36%)	90.98% (9.05%)	10.94% (7.19%)	90.87% (8.88%)	93.78% (5.89%)
Liver cancer	84.23% (12.81%)	67.66% (18.69%)	19.38% (8.65%)	68.73% (16.87%)	82.58% (7.60%)
Melanoma	88.05% (22.11%)	87.05% (17.09%)	27.60% (21.71%)	87.07% (16.67%)	89.31% (7.84%)
Obesity	78.45% (23.95%)	84.80% (15.10%)	13.25% (10.25%)	84.71% (14.68%)	82.18% (16.02%)

Sensitivity = TP/(TP + FN), specificity = TN/(TN + FP), precision = TP/(TP + FP), and accuracy = (TP + TN)/(TP + FP + FN + TN); where TP (true positive) is a true included citation identified as an include or no decision, FN (false negative) is a true included citation identified as an exclude with a reason for exclusion, TN (true negative) is a true excluded citation identified as an exclude with a reason for exclusion, and FP (false positive) is a true excluded citation identified as an include or identified as having no decision
^aCorrect reason for exclusion was defined as the number of citations whose true reason for exclusion fell above the 90% threshold over the total number of citations with any reason for exclusion. Sensitivity, specificity, precision, and accuracy were calculated by holding each factor constant while averaging over all other model characteristics (e.g., downsampling and performance metric)
Abbreviations: SD, standard deviation

ISSN: 2046-4053