Compare and diagnose

Run one frozen comparison

ragvi compare-data ./docs --questions ./questions.txt --techniques bm25,parent-child
ragvi compare-data ./docs --reviewed ./reviewed.json --techniques bm25,parent-child,compress --k 1 --chunk-size 100 --chunk-overlap 10

Inputs load once. Without reviewed references the comparison is unscored. With references, evidence gained/lost/missing and per-question outcomes accompany aggregate metrics. A neutral result or keeping the baseline is valid.

Suites and controlled grids

{"schema_version":1,"baseline":"body","policy":"fixed_k","variants":[{"name":"body","technique":"bm25"},{"name":"parents","technique":"parent-child"}]}

ragvi compare-data ./docs --reviewed ./reviewed.json --suite ./suite.json

Suite variants can set explicit stage options and sidecars. Grids are bounded before execution: default 16, maximum 32 expanded configurations; 0–3 extra timing trials; 100–10,000 bootstrap resamples. Query-call limits exclude indexing, downloads and provider-internal work, so are not billing limits.

Uncertainty and resources

Seeded 95% percentile intervals resample paired question differences, optionally grouping related questions. Fewer than two groups gives an N/A interval. Small samples and multiple-comparison limits apply; repeated timing runs do not add independent questions. Timing/cache observations are labeled; missing measurements stay unavailable.

Optional criteria report matched, not-matched or unknown. Cost estimates require supplied usage and identified pricing; they are not provider bills or a fabricated saving.

Failures and resume

ragvi compare-data . --resume ./ragvi-reports/COMPARISON --format html --format json

Replace COMPARISON with the printed directory. Resume validates frozen digests, settings, contexts, checkpoints and recomputed evidence before reusing completed work. It rereads frozen inputs rather than INPUT. Failed trials get new attempts and previous files remain. Configuration changes require a fresh comparison. Any participant failure produces a partial-failure report and nonzero status; a failed baseline is not silently replaced.

• Suites, heldout shortlists, criteria and resume contracts