Evaluation
Both frozen versions, scored on the same 61 held-out items from EXP-002. The frozen weights are used as released and never refit. Each chart uses 61 items, so small differences between versions are within noise.
Reliability diagrams
When the model says 80%, is it right about 80% of the time? Points on the dashed line mean yes. Each dot is a group of items with similar predictions, and bigger dots hold more items. Temperature scaling (Guo et al. 2017) is what moves the points toward the line.
Confusion matrices
Rows are the editor's answer, columns the model's, at a 50% threshold. Red cells are mistakes.
| Model: style | Model: not style | |
|---|---|---|
| Editor: style | 34 | 2 |
| Editor: not style | 6 | 19 |
Confident and right 35, confident and wrong 0, not sure 26.
| Model: style | Model: not style | |
|---|---|---|
| Editor: style | 33 | 3 |
| Editor: not style | 3 | 22 |
Confident and right 44, confident and wrong 2, not sure 15.
Confidence histograms
How the predicted probabilities spread across the test items. Red is items the editor labeled style, grey is not style. Items piled at the edges are confident. Items in the middle are the not-sure ones.
Coverage against confidence
Conformal prediction trades coverage for confident answers through one setting, α. A lower α promises the right answer more often and gives more not-sure answers. Released versions use α = 0.10.
| α | Target | v0.0.1 coverage | v0.0.1 confident | v0.0.2 coverage | v0.0.2 confident |
|---|---|---|---|---|---|
| 0.05 | 95% | 100% | 36% | 98% | 59% |
| 0.10 | 90% | 100% | 57% | 97% | 75% |
| 0.15 | 85% | 89% | 97% | 92% | 93% |
| 0.20 | 80% | 82% | 93% | 85% | 93% |
| 0.25 | 75% | 82% | 90% | 82% | 89% |
| 0.30 | 70% | 77% | 82% | 82% | 89% |
Thresholds recomputed for each α on the 45 calibration labels.
Learning curve (exploratory)
Fresh models trained on growing numbers of random labels, with the released design, 20 random subsets per size. The two lines are labeled on the chart. Shaded bands span the 20 subsets, and the red ring marks the 82% reached with 35 hard cases in EXP-002. This analysis was not pre-registered, and these are not released models.
Computed 2026-09-26 by src/ari3_charts.py in 17.9 s (Python 3.13.2, scikit-learn 1.9.1).