Table 5 Semi-automation test with RobotAnalyst using a training set of dually reviewed citations from a review-of-review with labels from full-text screening
| Â | Traditional search Total citations: 3181 Title-abstract screening: 201 includes/2980 excludes Full-text screening: 59 includes/3122 excludes | |||
|---|---|---|---|---|
| Â | Review of reviews Total SR citations: 30 Total primary study citations: 95 Title-abstract screening: 76 includes/49 excludes Full-text screening 33 includes/43 excludes Number of citations not screened: 3030 | Training set: ROR citations Labeled citations: 125 (4%) Training set labels: TP (33), FP (0), TN (92) Unlabeled citations assigned inclusion prediction by ML algorithm: 3056 | ||
| Â | Â | Inclusion prediction: 0.3 | Inclusion prediction: 0.4 | Inclusion Prediction: 0.5 |
Predicted includes | NA | 3040 | 2819 | 1166 |
Predicted excludes | NA | 16 | 237 | 1890 |
Sensitivity | 54% | 100% | 97% | 69% |
Specificity | 100% | 3% | 10% | 63% |
Missed citations | 26 | 0 | 2 | 18 |
Burden | 4% | 99% | 93% | 41% |
Time savings (min) | 1561 | 8 | 118.5 | 945 |