Table 6 Semi-automation test with RobotAnalyst using a training set of dually reviewed citations from a review-of-review and randomly selected citations with labels from full-text screening
| Â | Traditional search Total citations: 3181 Title-abstract screening: 201 includes/2980 excludes Full-text screening: 59 includes/3122 excludes | ||
|---|---|---|---|
| Â | Training set: ROR citations + 30% random citations Labeled citations: 1063 (33%) Training set labels: TP (40), FP (0), TN (1023) Unlabeled citations assigned inclusion prediction by ML algorithm: 2118 | ||
Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion prediction: 0.5 | |
Predicted includes | 2094 | 1765 | 676 |
Predicted excludes | 24 | 353 | 1442 |
Sensitivity | 98% | 84% | 80% |
Specificity | 34% | 44% | 79% |
Missed citations | 1 | 3 | 12 |
Burden | 99% | 89% | 55% |
Time savings (min) | 12 | 177 | 721 |