Table 4 Semi-automation test with RobotAnalyst using a training set of dually reviewed randomly selected citations with labels from title and abstract screening
| Â | Traditional database search Total citations: 3128 Title-abstract screening: 148 includes/2980 excludes Full-text screening: 46 includes/3082 excludes | ||
|---|---|---|---|
Training set Labeled citations: 938 (30%) Training set labels: TP (15), FP (32), TN (891) Unlabeled citations assigned inclusion prediction by ML algorithm: 2190 | |||
Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion prediction: 0.5 | |
Predicted includes | 2168 | 1970 | 1363 |
Predicted excludes | 22 | 220 | 827 |
Sensitivity | 100% | 93% | 74% |
Specificity | 30% | 36% | 55% |
Missed citations | 0 | 3 | 12 |
Burden | 99% | 93% | 74% |
Time savings (min) | 11 | 110 | 413.5 |