EXP-003

Summary

EXP-003 asks whether the evidence ARI3 produces can be trusted, in two parts. EXP-003A tested frozen v0.0.2 on articles published after it was frozen. EXP-003B trained a new perception head to tell a prediction from a report.

EXP-003A ran once on 150 items published after the freeze and labeled by the editor. The prediction sets held the right answer for 0.927 of them and the not-sure share was 0.327, so H1 and H3 were supported. Accuracy was 0.833 against a pass mark of 0.85, so H2 was not supported. The model was not refit, recalibrated or retuned, and the evaluation was not rerun.

EXP-003B ran once, as pre-registered. H4 failed: the new head answered "not a prediction" for every test item, so its balanced accuracy was 0.50, the same as always answering no. H5 held: coverage was 0.961.

The failed forecast head is frozen and hashed, and it is not used. Forecast articles stay in the mention counts. v0.0.3 was released on 2026-10-03 with a release manifest. The style classifier is still the frozen v0.0.2 model.

Metadata

Experiment

EXP-003 (EXP-003A Temporal Generalization, EXP-003B Forecast Detection)

Pre-registration

2ff1c53, 2026-09-26 22:30 UTC

Pre-registration hash

sha256 fa65367d3166cba52903229bfd849ce34022b536e93ae6c62341325e7911aff1

EXP-003A run

2026-10-02, once

EXP-003A record

1c78f56

EXP-003A dataset

150 time-holdout labels by the editor (82 yes, 68 no), frozen and committed before the run

EXP-003A gold fingerprint

sha256 7e15f2a1c9595fce12dffd5558b9e38e1845a32a22052b0664649235676d9b7d

EXP-003A model

ari3-v0.0.2, unchanged. Weights sha256 f6b25eb421e2766cc2681f9b2a6816330ab50569bae752a73b231ed7fe282f09

EXP-003A eligibility

Amended 2026-09-30, before any label existed

EXP-003B run

2026-09-26, once

EXP-003B record

bbbaaf0

EXP-003B dataset

250 is_forecast labels by the editor (35 forecasts), made after the pre-registration

Dataset fingerprint

sha256 ff36351a44866144506dddf9cf58fdf299dd61100ed49238c534f2d75633ec70

is_forecast head

Frozen and hashed, not used. manifest_sha256 c39b1f601e7d2fa137a48532ceddea38bb7230cc2b7e0976e4ffcb60b93a6d77

Release manifest

2026-10-03, manifest_sha256 846b249dfaa46c0aec920998d8f83e0cc50c10de7974793516fa60fcd894ebc1

Environment

Recorded in the forecast head's manifest and in the EXP-003A result file

Random seeds

EXP-003B queue: 11. EXP-003A queue: 13. Model fitting is deterministic.

Research question

Can the evidence ARI3 produces be trusted: do its judgments hold on articles it could not have seen (EXP-003A), and can it separate predictions from reports (EXP-003B)?

Pre-registered: Yes. Both parts, their five hypotheses, a reason for each threshold and a failure interpretation for each were committed before any EXP-003 label existed.

Hypotheses and results

Why these hypotheses

Every earlier test used articles collected before the model was trained. In use, the model meets articles that did not exist when it was frozen, so only post-freeze data shows how it will behave. v0.0.2 answered not-sure for 25% of its test set but 42% of the whole corpus, a first sign that this matters.

An article saying a look "will be big next season" is a prediction. Counted as a mention, it feeds the press's forecasts back into the evidence that later trend models will read. A forecast detector has to exist before those models run.

Method

1. 003A: 150 items published after the v0.0.2 freeze, drawn in turn from each sector
   ↓
2. 003A: labeled by the editor, then frozen and committed
   ↓
3. 003A: frozen v0.0.2 scored them once, unchanged
   ↓
4. 003B: 250 is_forecast labels, 150 random and 100 containing a forecast word
   ↓
5. 003B: split fixed by a hash of each item, 161 train, 38 calibration, 51 test
   ↓
6. 003B: new logistic regression head on the shared, frozen sentence embeddings
   ↓
7. 003B: temperature scaling, then a split conformal threshold at α = 0.10
   ↓
8. 003B: scored on 51 test items, both groups together and separately

Experimental controls

Results

EXP-003A: frozen v0.0.2 scored once on the 150 time-holdout items. EXP-003B: the is_forecast head and always answering no, scored on the same 51 held-out items.

Cost of improvement

Surprising findings

Threats to validity

• 5 forecasts in the EXP-003B test set, so every forecast metric is highly uncertain.
• One editor labels every item, with no second labeler.
• The 100 forecast-word items are not a random sample.
• The EXP-003A sample covers 39 outlets and about 3.3 days of publication, so a later period may behave differently.
• EXP-003A measures how v0.0.2 does on newer articles. It does not explain why accuracy fell.
• Publish times are the times feeds report, and a feed can report them wrongly.
• 88.6% of stored items came from the editorial sector on 2026-10-03. The EXP-003A draw rotates through sectors, so its sample is less editorial than the corpus.
• Most labeled items are tagged English.

Engineering changes

• Review app runs any labeling queue, each with its own task, split and group tag
• Queue builders for the forecast sample and the time holdout (src/label_tool.py)
• One-shot EXP-003B runner that refuses to run twice (src/ari3_exp003b_run.py)
• EXP-003A runner that checks every label and eligibility rule, freezes the gold labels, and refuses to score before they are committed and public (src/ari3_exp003a_run.py)
• Database migration 0004: write-once first-seen times, so a holdout can show when each item arrived
• Manifests now record the environment, runtime and labeling queue fingerprint

Research log

1. ObservationPress predictions were being counted as mentions of the looks they predicted.
   ↓
2. IdeaAdd a second perception task that tells a prediction from a report.
   ↓
3. ExperimentEXP-003B (250 labels) and EXP-003A (150 time-holdout labels), pre-registered, each run once.
   ↓
4. Result003B: H4 failed, the head never answered yes, and H5 held. 003A: coverage and the not-sure share held, and accuracy missed its pass mark.
   ↓
5. New questionWhich newer articles does v0.0.2 get wrong, and would labels from newer periods fix them? Any fix is a new pre-registered experiment.

Lessons

• Expected: the style task's design would transfer to forecasts. Observed: with forecasts at 11% of labels, it never predicted one.
• Plain accuracy looked like 90% success. Balanced accuracy showed it was no better than always answering no.
• A failed hypothesis still produced a usable record: what failed, the observed failure pattern, and a frozen head that is kept out of the counts.
• Expected: v0.0.2 would keep its accuracy on newer articles. Observed: accuracy fell to 0.833 while coverage held at 0.927.

Open questions

1. Which kinds of newer articles does v0.0.2 get wrong, and why?
2. How many forecast labels would a head need before it separates predictions from reports on new items?
3. Would more context than a headline and excerpt make forecasts easier to recognize?

Reproducibility

The pre-registration, both runners, the frozen forecast head, the EXP-003A gold labels (item IDs and labels) and the EXP-003A result are public. The items' text stays in the project's local database, so outsiders can check the hashes but cannot rerun the scoring. Both runners refuse to run a second time.

git clone https://github.com/arikthehacker/fashion-trend-crawler.git
cd fashion-trend-crawler
git show 2ff1c53:models/ari3-v0.0.3/PREREGISTRATION.md | sha256sum
git show bbbaaf0:models/ari3-v0.0.3/is_forecast/weights.npz | sha256sum
git show 88eb59e:models/ari3-v0.0.3/exp003a/gold_v1.jsonl | sha256sum
git show 1c78f56:models/ari3-v0.0.3/exp003a/result.json | sha256sum