# Controlled experiments

Freeze inputs, isolate decisions and protect heldout questions.

CLI version: 0.2.1 · [Read online](<https://ragvidence.medo-id.com/docs/controlled-experiments>)

## Change one meaningful variable

Compare a specific decision using the same original corpus, reviewed references and questions. Keep final depth or context policy explicit. If you change chunking, model, k and labels together, the result describes the combined intervention.

## Dev exploration and heldout evaluation

Reviewed comparisons default to --phase dev: heldout questions are not retrieved or scored. A successful comparison writes shortlist.json with frozen configurations. --shortlist-names selects a subset including the baseline; the CLI does not optimize the choice.

**Terminal**

```
ragvi compare-data ./docs --reviewed ./reviewed.json --techniques bm25,parent-child
ragvi compare-data ./docs --phase heldout --shortlist ./ragvi-reports/COMPARISON/shortlist.json
```

Replace COMPARISON with the printed directory, keep its frozen files together, and include heldout questions in your own references. The downloadable four-question website example has only dev questions; it is not a heldout benchmark.

## Heldout reuse disclosure

Reuse is recorded locally under the chosen output root across shortlists for the same corpus/references. Moving/deleting that ledger defeats the disclosure. Public or repeatedly tuned heldout data is not unseen evaluation data.

## Preserve provenance

Keep configurations, source/reference hashes, provider/model identities, warnings and stage evidence. Resuming validates frozen artifacts; make a fresh comparison when the experiment changes.