Controlled experiments

Change one meaningful variable

Compare a specific decision using the same original corpus, reviewed references and questions. Keep final depth or context policy explicit. If you change chunking, model, k and labels together, the result describes the combined intervention.

Dev exploration and heldout evaluation

Reviewed comparisons default to --phase dev: heldout questions are not retrieved or scored. A successful comparison writes shortlist.json with frozen configurations. --shortlist-names selects a subset including the baseline; the CLI does not optimize the choice.

ragvi compare-data ./docs --reviewed ./reviewed.json --techniques bm25,parent-child
ragvi compare-data ./docs --phase heldout --shortlist ./ragvi-reports/COMPARISON/shortlist.json

Replace COMPARISON with the printed directory, keep its frozen files together, and include heldout questions in your own references. The downloadable four-question website example has only dev questions; it is not a heldout benchmark.

Heldout reuse disclosure

Reuse is recorded locally under the chosen output root across shortlists for the same corpus/references. Moving/deleting that ledger defeats the disclosure. Public or repeatedly tuned heldout data is not unseen evaluation data.

Preserve provenance

Keep configurations, source/reference hashes, provider/model identities, warnings and stage evidence. Resuming validates frozen artifacts; make a fresh comparison when the experiment changes.