EXP-001

Summary

ARI3 v0.0.1 is the first model trained on the editor's own judgments, 145 labels saying whether a news item is about style.

It pairs a small classifier with calibrated probabilities and conformal prediction, so it can say when it is not sure.

It reached 0.882 held-out accuracy against a 0.529 baseline, and all 4 of its mistakes came with a not-sure answer.

Metadata

Experiment

EXP-001

Run

2026-09-23, frozen 08:55 UTC

Release commit

9c296bd

Dataset

145 random labels by the editor

Dataset fingerprint

sha256 3f09bf6fe74b66102b4a154d5a75eb607f71292fc695cb84155bebb0c2a3978f

Model

ari3-v0.0.1 (perception, is_style_signal)

Weights

sha256 78ddaf3383eaa141687133e8d6993ff12a6568452a1747bd862c0955572b76be

Manifest

manifest_sha256 f9f2c44bb06dad0fc3672c29dfcacfa39d2ce503dc28c38457206dac5ccb0213

Embedder

paraphrase-multilingual-MiniLM-L12-v2 @ e8f8c21

Environment

Python 3.13.2, numpy 2.5.0, scikit-learn 1.9.1, sentence-transformers 6.1.0

Random seeds

Label queue: 7. Model fitting is deterministic.

Runtime

Not recorded.

Research question

Can a small model trained on one editor's labels tell style coverage from adjacent news, and know when it is unsure?

Pre-registered: No. EXP-001 was exploratory. Its result became the baseline that EXP-002 was pre-registered against.

Method

1. 145 labels by the editor
   ↓
2. Split fixed by a hash of each item: 82 train, 29 calibration, 34 test
   ↓
3. Multilingual sentence embeddings, frozen
   ↓
4. Logistic regression
   ↓
5. Temperature scaling
   ↓
6. Split conformal threshold at α = 0.10
   ↓
7. Scored on 34 held-out items

Experimental controls

Results

Scored on held-out items the model never trained on.

Cost of improvement

Surprising findings

Threats to validity

• Very small test set. 34 items, so accuracy has a 22-point interval.
• One annotator, one pass.
• Source bias. Editorial sources already dominated the stored corpus when EXP-001 ran.
• Only 29 calibration items, which makes the conformal threshold cautious.
• Not pre-registered. The analysis was chosen by the people who ran it.

Engineering changes

• Database migration 0002: feed excerpts, a labels table with hash-fixed splits, and an append-only prediction log
• Keyboard labeling tool (src/label_tool.py)
• Freeze script that refuses to overwrite a frozen version (src/ari3_freeze.py)

Research log

1. ObservationFeeds mix style coverage with business and celebrity news, and raw counts cannot tell them apart.
   ↓
2. IdeaTrain a small model on the editor's own judgment of what counts as style.
   ↓
3. ExperimentEXP-001, 145 labels, calibrated and conformal.
   ↓
4. Result0.882 accuracy and no confident mistakes, but many not-sure answers.
   ↓
5. New questionCan hard examples make it more decisive? That became EXP-002.

Lessons

• Expected: feed counts could stand in for style activity. Observed: style share varies nearly fourfold by sector.
• On the test set, the not-sure answer flagged every one of the model's mistakes for the editor.

Open questions

1. Where does the model's boundary sit, and which items sit on it?
2. Is beauty a separate category? (Answered before EXP-002: yes, with nails counted as style.)

Reproducibility

The weights and manifest are public. The labels are in the project's local database and not published yet. The freeze script refuses to refit a frozen version, so a rerun needs a new version name.

git clone https://github.com/arikthehacker/fashion-trend-crawler.git
cd fashion-trend-crawler
git show 9c296bd:models/ari3-v0.0.1/weights.npz | sha256sum