Claim ledger

This page lists the claims approved for publication from ARI3's claim ledger, with the committed artifact that supports each one. A published claim is not silently rewritten or deleted. If its status changes, the later record preserves that change. Failed pre-registered hypotheses remain visible as not supported.

The repository ledger is the underlying record. This page publishes only claim IDs explicitly approved for the site. The ledger file is at CLAIM_LEDGER.md.

1. C-0001

   MeasuredEXP-001 · as of 2026-09-23

   Frozen ARI3 v0.0.1 (is_style_signal) classified its held-out test items with accuracy 0.882.

   Result

   0.882 (30 of 34 correct). Conformal coverage 1.000 on the same 34 items.

   Limitations

   n = 34. EXP-001 had no pre-registered criterion. The same model scored 0.869 on EXP-002's 61-item test split (C-0004), a different set of items.

   Artifact

   models/ari3-v0.0.1/manifest.json, metrics_test.test_accuracy (manifest SHA-256 f9f2c44b…)

   First valid commit

   9c296bd

2. C-0002

   SupportedEXP-002 · as of 2026-09-26

   EXP-002 H1: frozen v0.0.2 gave a single-label (decisive) conformal answer on at least 70% of test items.

   Result

   0.754 (n = 61), up from 0.574 for v0.0.1 on the same items, against a threshold of 0.70

   Limitations

   n = 61, a single test split. A decisive answer is a single-label conformal set, which can still be wrong.

   Artifact

   models/ari3-v0.0.2/manifest.json, hypotheses.H1 (manifest SHA-256 d1cad506…)

   First valid commit

   ec788a0

3. C-0003

   SupportedEXP-002 · as of 2026-09-26

   EXP-002 H2: frozen v0.0.2's conformal sets covered the true label on at least 90% of test items.

   Result

   0.967 (n = 61), against a threshold of 0.90

   Limitations

   n = 61. Coverage is guaranteed only on average over exchangeable data. EXP-003A tests it on later items.

   Artifact

   models/ari3-v0.0.2/manifest.json, hypotheses.H2

   First valid commit

   ec788a0

4. C-0004

   SupportedEXP-002 · as of 2026-09-26

   EXP-002 H3: v0.0.2 was at least as accurate as v0.0.1 on the same test items.

   Result

   0.902 vs 0.869 (n = 61). 95% intervals [0.802, 0.954] and [0.762, 0.932]

   Limitations

   The intervals overlap heavily. The result supports "no worse" and does not show a difference in accuracy.

   Artifact

   models/ari3-v0.0.2/manifest.json, hypotheses.H3 and results_test

   First valid commit

   ec788a0

5. C-0005

   SupportedEXP-002 · as of 2026-09-26

   EXP-002 H4: v0.0.2's expected calibration error after temperature scaling was at most 0.10.

   Result

   0.025 (n = 61), against a threshold of 0.10. Before scaling: 0.152.

   Limitations

   ECE on 61 items with 5 bins is noisy

   Artifact

   models/ari3-v0.0.2/manifest.json, hypotheses.H4

   First valid commit

   ec788a0

6. C-0006

   SupportedEXP-002 · as of 2026-09-26

   EXP-002 H5: training with active-learning labels gave higher test accuracy than an equal number of random labels.

   Result

   Mean 0.898 over 20 subsets (range 0.869 to 0.918) vs 0.885 for the random arm. Arm size 194, of which 35 active-learning labels.

   Limitations

   A difference of 0.013 on 61 test items is less than one item. The subsets vary more than the gap between arms.

   Artifact

   models/ari3-v0.0.2/manifest.json, hypotheses.H5 and h5

   First valid commit

   ec788a0

7. C-0007

   Not supportedEXP-003B · as of 2026-09-26

   EXP-003B H4: the is_forecast head reached balanced accuracy of at least 0.75 and beat the majority baseline.

   Result

   Balanced accuracy 0.50, equal to the majority baseline (n = 51, 5 forecasts). The head labeled every test item "not a forecast".

   Limitations

   5 positive test items

   Artifact

   models/ari3-v0.0.3/is_forecast/manifest.json, hypotheses.H4 (manifest SHA-256 c39b1f60…). models/ari3-v0.0.3/AMENDMENT_2026-09-26_findings.md

   First valid commit

   bbbaaf0

8. C-0008

   SupportedEXP-003B · as of 2026-09-26

   EXP-003B H5: the is_forecast head's conformal sets covered the true label on at least 90% of test items.

   Result

   0.961 (n = 51), against a threshold of 0.90

   Limitations

   Coverage was met while the head detected no forecasts (C-0007), so this claim says nothing about forecast detection.

   Artifact

   models/ari3-v0.0.3/is_forecast/manifest.json, hypotheses.H5

   First valid commit

   bbbaaf0

9. C-0010

   MeasuredEXP-004 · as of 2026-10-01

   On the frozen EXP-004 held-out TEST set, ARI3's frozen hybrid retriever achieved macro Recall@10 = 0.770 and Hit@10 = 1.000 across the 12 TEST questions with at least one resolved human-relevant item. The retriever was selected with a pre-registered DEV rule, frozen before TEST, and evaluated on TEST once.

   Result

   Recall@10 0.770, Recall@5 0.438, Hit@10 1.000, Hit@5 1.000, MRR 0.958, Unsure@10 0.311 (18 questions with results). Latency median 67.4 ms, p95 344.4 ms. 0 integrity failures. 19 TEST questions in total, 7 of them without a resolved relevant item.

   Limitations

   Only 12 answerable TEST questions carry the primary metrics. Relevance is relative to the frozen judged pool (the union of each candidate's top 10), not to every relevant item in the corpus. Multilingual retrieval performance is not established: most non-English judgments are unsure, and no non-English TEST question has a resolved relevant item. Some filtered questions have small candidate sets (10 to 16 eligible items), where high recall is close to guaranteed. Recall@10 is capped for questions with more than 10 relevant items (two TEST questions reached the cap). Feed language tags contain known errors. This does not mean that ARI3 retrieves 77% of all relevant evidence.

   Artifact

   experiments/exp-004-grounded-retrieval/results/test-retrieval-v1.json (metrics.all), frozen retriever frozen/retriever_v1.json (SHA-256 3b55fceb…), selection rule frozen/retriever_selection_rule_v1.json (111cd0f4…) and amendment 1 (01d9c1d0…)

   First valid commit

   b9f9e6c

10. C-0011

    SupportedEXP-003A (ARI3 v0.0.3 INTEGRITAS) · as of 2026-10-02

    Frozen ARI3 v0.0.2 kept its conformal coverage guarantee on 150 items published after it was frozen: the true label was inside the prediction set for 0.927 of them, against a pre-registered criterion of at least 0.90.

    Result

    coverage 0.927 (139 of 150). Prediction sets: 57 {yes}, 44 {no}, 49 {yes, no}, 0 empty.

    Limitations

    One editor labeled every item, with no second labeler. 150 items published 2026-09-26 20:00 to 2026-09-30 04:03 UTC (about 3.3 days after the freeze), from 39 outlets, mostly editorial. Domain shift is measured, not explained. published_at is feed-reported. Model and decision rule unchanged: no refit, recalibration or threshold change.

    Artifact

    models/ari3-v0.0.3/exp003a/result.json (SHA-256 aab0ca4a…), gold exp003a/gold_v1.jsonl (7e15f2a1…), pre-registration models/ari3-v0.0.3/PREREGISTRATION.md (fa65367d…, commit 2ff1c53), eligibility amendment of 2026-09-30 (32690a8b…)

    First valid commit

    1c78f56 (gold frozen in 88eb59e before the run)

11. C-0012

    Not supportedEXP-003A (ARI3 v0.0.3 INTEGRITAS) · as of 2026-10-02

    Frozen ARI3 v0.0.2 reached accuracy 0.833 on 150 items published after it was frozen. The pre-registered criterion was at least 0.85, so H2 is not supported. Its accuracy on its own test set was 0.902 (C-0004).

    Result

    accuracy 0.833 (125 of 150, Wilson 95% interval 0.766 to 0.884). Precision 0.861, recall 0.829 for style items. 25 errors, 11 of them confident (a single-label set that excludes the true label).

    Limitations

    One editor labeled every item, with no second labeler. 150 items published 2026-09-26 20:00 to 2026-09-30 04:03 UTC (about 3.3 days after the freeze), from 39 outlets, mostly editorial. Domain shift is measured, not explained. published_at is feed-reported. Model and decision rule unchanged: no refit, recalibration or threshold change.

    Artifact

    models/ari3-v0.0.3/exp003a/result.json (SHA-256 aab0ca4a…), gold exp003a/gold_v1.jsonl (7e15f2a1…), pre-registration models/ari3-v0.0.3/PREREGISTRATION.md (fa65367d…, commit 2ff1c53), eligibility amendment of 2026-09-30 (32690a8b…)

    First valid commit

    1c78f56 (gold frozen in 88eb59e before the run)

12. C-0013

    SupportedEXP-003A (ARI3 v0.0.3 INTEGRITAS) · as of 2026-10-02

    On 150 items published after it was frozen, frozen ARI3 v0.0.2 answered "not sure" ({yes, no}) for 0.327 of them, within the pre-registered limit of 0.35. The share lies between its test-set rate (0.25) and its whole-corpus rate (0.42).

    Result

    not-sure share 0.327 (49 of 150). Decisive share 0.673.

    Limitations

    One editor labeled every item, with no second labeler. 150 items published 2026-09-26 20:00 to 2026-09-30 04:03 UTC (about 3.3 days after the freeze), from 39 outlets, mostly editorial. Domain shift is measured, not explained. published_at is feed-reported. Model and decision rule unchanged: no refit, recalibration or threshold change.

    Artifact

    models/ari3-v0.0.3/exp003a/result.json (SHA-256 aab0ca4a…), gold exp003a/gold_v1.jsonl (7e15f2a1…), pre-registration models/ari3-v0.0.3/PREREGISTRATION.md (fa65367d…, commit 2ff1c53), eligibility amendment of 2026-09-30 (32690a8b…)

    First valid commit

    1c78f56 (gold frozen in 88eb59e before the run)