EXP-002

Summary

ARI3 v0.0.2 kept every design choice of v0.0.1 and changed only the training data, adding 155 random labels and 50 hard cases chosen by active learning.

The plan and its five pass criteria were committed publicly before any v0.0.2 label existed, so the result could not be tuned after the fact.

All five hypotheses were supported. The clear gains were in confident answers and calibration, accuracy moved within noise, and the model made 2 confident mistakes where v0.0.1 made none.

Metadata

Experiment

EXP-002

Run

2026-09-26 19:25 UTC

Pre-registration

79e95bc, 2026-09-24 01:08 UTC

Release commit

ec788a0

Dataset

350 labels by the editor: 300 random, 50 active learning

Dataset fingerprint

sha256 20ca8e4a1344f81e6c38527935c57542cfa9ff39f61fbb4981560abf10698765

Model

ari3-v0.0.2 (perception, is_style_signal)

Weights

sha256 f6b25eb421e2766cc2681f9b2a6816330ab50569bae752a73b231ed7fe282f09

Manifest

manifest_sha256 d1cad506d6c9326e77a7729970261a0bb8a0ace3e834b0735530a98e0214e055

Embedder

paraphrase-multilingual-MiniLM-L12-v2 @ e8f8c21

Environment

Not recorded in this manifest. The run used the same machine as EXP-001. Future manifests record it.

Random seeds

Label queue: 7. H5 subsets: 0 to 19. Model fitting is deterministic.

Runtime

Not recorded. Future manifests record it.

Research question

Does adding intentionally difficult human labels improve perception more than labeling the same number of randomly chosen articles?

Pre-registered: Yes. Five hypotheses with pass criteria were committed before any v0.0.2 label existed.

Hypotheses and results

Why these hypotheses

A classifier learns where to draw the line between two answers. Items far from that line teach it little, because it already gets them right. Items the model is torn about sit near the line, so labeling them shows where the line belongs. Choosing those items on purpose is uncertainty sampling, a standard active-learning method (Settles 2009).

More calibration labels should also help. Split conformal prediction pads its threshold to guarantee coverage on small calibration sets, and the padding shrinks as the set grows. A smaller threshold means fewer not-sure answers without breaking the guarantee (Angelopoulos & Bates 2021).

Method

1. 350 labels by the editor
   ↓
2. Split fixed by a hash of each item: train, calibration, test
   ↓
3. Hard cases removed from calibration and test
   ↓
4. Multilingual sentence embeddings, frozen
   ↓
5. Logistic regression on 229 training labels
   ↓
6. Temperature scaling on 45 calibration labels
   ↓
7. Split conformal threshold at α = 0.10
   ↓
8. Scored with v0.0.1 on the same 61 test items

Experimental controls

Results

v0.0.1 and v0.0.2 scored on the same 61 held-out items.

Cost of improvement

Surprising findings

Threats to validity

• Small test set. 61 items, so a 2-item difference is within noise.
• One annotator. Every label reflects one editor's judgment, and no agreement rate between labelers exists yet.
• Source bias. About 87% of stored items came from the editorial sector when EXP-002 ran (2026-09-26), so small sectors had few examples.
• Language. By feed metadata, 300 of the 350 labeled items are tagged English and 10 French, so accuracy on other languages is essentially untested.
• Time. 276 of the 350 labeled items were published in September 2026, so the model is barely tested on older or newer language.
• Class balance. 55% of random labels are yes.
• The test set excludes hard cases, so it does not measure performance on the hardest items.
• Two analysis choices were not in the pre-registration. Hard cases were also kept out of calibration, and H5 was run at equal size by swapping labels. Both are listed in the manifest.

Engineering changes

• Touch and keyboard labeling app with a separate hard-case deck (src/ari3_review.pyw)
• Hard-case labels are tagged in the database so they can be kept out of testing
• One-shot experiment runner that refuses to run twice (src/ari3_v002_run.py)
• Scope rule for beauty coverage written into the labeling screen

Research log

1. Observationv0.0.1 answered not-sure for 1,648 of 2,472 unlabeled items.
   ↓
2. IdeaThe items it is most torn about should teach it the most.
   ↓
3. ExperimentEXP-002, pre-registered, 50 hard cases plus 155 random labels.
   ↓
4. ResultConfident answers rose clearly. Accuracy moved within noise. 2 confident mistakes appeared.
   ↓
5. New questionDoes the confident-mistake rate grow as the model becomes more decisive?

Lessons

• Expected: hard cases would mainly raise accuracy. Observed: they mainly raised confidence.
• Decisiveness has a price, and it has to be reported beside the gain.
• Pre-registration required the weak results to be published.

Open questions

1. How do accuracy and confident answers change as labels grow (a learning curve)?
2. Should the confident-mistake rate become a pass criterion of its own?
3. Does performance hold on items newer than the training data?
4. Does a second labeler agree with the first, and how often?

Reproducibility

The code, weights and manifest are public. The labels live in the project's local database and are not published yet, so an outside rerun is not possible yet. The published hashes let anyone confirm the weights are the ones committed.

git clone https://github.com/arikthehacker/fashion-trend-crawler.git
cd fashion-trend-crawler
git show ec788a0:models/ari3-v0.0.2/weights.npz | sha256sum
# with the label database present:
git checkout ec788a0
pip install -r requirements.txt -r requirements-ml.txt
python src/ari3_v002_run.py