EXP-004

Summary

EXP-004 asks whether ARI3 can find the stored records that answer a question, inside the question's date, language, sector and outlet limits. Each record is a headline and the feed's own summary, capped at 500 characters. No full article text is stored, so EXP-004 measures retrieval over ARI3's stored evidence, not over complete articles.

Four retrieval methods were compared on 37 development questions under a selection rule committed before any result. The hybrid method, which combines keyword search and sentence embeddings by reciprocal rank fusion, was chosen and frozen, then run once on 19 held-out test questions.

On the 12 test questions with at least one item the editor judged relevant, macro Recall@10 was 0.770 and Hit@10 was 1.000. This does not mean ARI3 finds 77% of all relevant evidence. Recall is measured against the pool of items the methods retrieved and the editor judged.

Metadata

Experiment

EXP-004. It does not imply ARI3 v0.0.4

Closed

2026-10-01

Ledger claim

C-0010

Questions

56, drafted and reviewed by the editor, frozen 2026-09-30. 37 development, 19 test (seed 4)

Questions fingerprint

sha256 d1d17cb4cfe34196b05724dd8623ad4c94ce9c69d00420ba9af3290cf73d4ad4

Relevance judgments

819 question and item pairs judged by the editor, blind to method, rank and score: 318 relevant, 273 not relevant, 228 unsure

Judgments fingerprint

sha256 bda481d21f0b551f985d6ba05caa08a704542ec6c64c5baf7896ad9523153822

Index

7,955 stored items first seen before 2026-09-30 08:00 UTC, from 94 outlets

Frozen retriever

44421a5, sha256 3b55fcebbb08ec54b87198ab4a18e2f1b2bd0b922c6677428f525ceeadfd6b21

Test result

b9f9e6c, sha256 bf44847928dc7fd53f7c646e0c158ba3005ba1a39f6d1b01a5a6937d7fc87d19

Embedder

paraphrase-multilingual-MiniLM-L12-v2 @ e8f8c21

Research question

Can ARI3 find the stored records that answer a question, within the question's date, language, sector and outlet limits?

Pre-registered: Partly. The rule for choosing a retriever on development questions, and the single run on test questions, were committed before any result. The rule was amended once, before the development run. EXP-004 had no pass criteria.

Method

1. 56 questions, reviewed and frozen by the editor
   ↓
2. Each method's top 10 pooled and judged by the editor, blind to method, rank and score
   ↓
3. Questions split into 37 development and 19 test
   ↓
4. Four methods compared on development questions: keyword search with two tokenizers, sentence embeddings, and a hybrid of the two
   ↓
5. Hybrid chosen by the committed rule and frozen
   ↓
6. Frozen hybrid run once on the test questions

Experimental controls

Results

The frozen hybrid retriever, run once on the 19 test questions. Hit, recall and reciprocal rank cover the 12 questions with at least one item judged relevant.

Cost of improvement

Surprising findings

Threats to validity

• Only 12 answerable test questions carry the main metrics.
• Recall is measured against the judged pool, the union of each method's top 10, not against every relevant item in the corpus.
• Recall@10 is capped for questions with more than 10 relevant items. Two test questions reached the cap.
• Retrieval in languages other than English is not established. Most non-English judgments are unsure, and no non-English test question has a resolved relevant item.
• Some filtered questions have small candidate sets, 10 to 16 eligible items, where high recall comes close to being guaranteed.
• One person judged relevance, and 228 of the 819 judgments are unsure.
• Feed language tags contain known errors.

Engineering changes

• Read-only retrieval over the SQLite store, with FTS5 keyword search and stored sentence embeddings (src/rag_*.py)
• A blind relevance-judging deck in the ARI3 review app
• Guards on the test run, which refuses to start without a frozen retriever and refuses to run twice

Research log

1. ObservationAnswering questions about the corpus needs the right stored records first, inside each question's limits.
   ↓
2. IdeaCombine keyword search and sentence embeddings, and choose between methods by a rule fixed in advance.
   ↓
3. ExperimentEXP-004: 56 frozen questions, 819 blind judgments, selection on development questions, one test run.
   ↓
4. ResultHybrid frozen. Test Recall@10 0.770 and Hit@10 1.000 on 12 answerable questions.
   ↓
5. New questionCan a language model answer from these records without claiming more than they say? That is EXP-005, in development.

Lessons

• Expected: one method would clearly win. Observed: hybrid led, but its lead over keyword search on development questions was within noise.
• Non-English relevance could not be judged reliably, so retrieval in other languages stays unmeasured.

Open questions

1. How does retrieval do in languages other than English, judged by someone who reads them?
2. How should the index be refreshed as the corpus grows? A refreshed index is a new retrieval version with its own check.

Reproducibility

The questions, judgments, split, frozen retriever and both results are public. The item store and index stay local, so outsiders can check the hashes but cannot rerun retrieval.

git clone https://github.com/arikthehacker/fashion-trend-crawler.git
cd fashion-trend-crawler
git show 44421a5:experiments/exp-004-grounded-retrieval/frozen/retriever_v1.json | sha256sum
git show b9f9e6c:experiments/exp-004-grounded-retrieval/results/test-retrieval-v1.json | sha256sum