# RAGvidence technique comparison
Input: docs
Status: complete · Policy: fixed_k · Phase: dev
Baseline: bm25
- Keeping the baseline is valid. Choose using these cases and your requirements.
- Equal k does not imply equal context volume; resources are reported separately.
- Intervals pair questions or families. Repeated timings are not extra samples.
- These observations do not prove what caused a particular failure.
## bm25
Status: complete
Evidence scope: exact_returned_spans
- Reviewed evidence is scored against the union of exact original-text spans.
dev/all: 3 / 4 answerable sufficient · Coverage: 0.875 · MRR: 1.0 · Precision: 1.0
heldout/all: 0 / 0 answerable sufficient · Coverage: N/A · MRR: N/A · Precision: N/A
Mean known query ms: 0.8010242509044474 (0 missing)
Mean context: 161.25 characters
Estimated supplied-usage cost: N/A N/A
### How long should I hold the reset button, and what power is required?
Rank 1 · manual.md · Source text · Original characters 23–116
## Reset pairing
Hold the reset button for eight seconds to clear pairing. The status light
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 1.016311001876602,
"hits": [
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 1,
"score": 2.2885582447052,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long should I hold the reset button, and what power is required?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.050705999456113204,
"hits": [
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 1,
"score": 2.2885582447052,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 160,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
## Warranty
The warranty lasts two years. Keep your receipt. Accidental water damage is excluded.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.09969899838324636,
"hits": [
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 1,
"score": 3.4753642082214355,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long is the warranty and what damage is excluded?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.028574999305419624,
"hits": [
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 1,
"score": 3.4753642082214355,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 161,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
## Battery care
Store the device in a cool, dry place. Charge it every three months when unused.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.06612499782931991,
"hits": [
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 1,
"score": 1.5058517456054688,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How often should I charge an unused device?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.026300000172341242,
"hits": [
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 1,
"score": 1.5058517456054688,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 164,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
## Delivery
Standard delivery takes five working days. Tracked delivery is available in Morocco.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.05724900256609544,
"hits": [
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 1,
"score": 3.3280820846557617,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "Where is tracked delivery available?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.02544800008763559,
"hits": [
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 1,
"score": 3.3280820846557617,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 160,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
# Orion device manual
## Reset pairing
Hold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.
## Warranty
The warranty lasts two years. Keep your receipt. Accidental water damage is excluded. Contact the support team before returning a device.
## Battery care
Store the device in a cool, dry place. Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.
## Delivery
Standard delivery takes five working days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel arrives.
Observed stages and selection evidence
[
{
"name": "children",
"elapsed_ms": 0.11351600187481381,
"hits": [
{
"source_unit_id": "child-0",
"rank": 1,
"score": 2.315727949142456,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-2",
"rank": 2,
"score": 1.4516243934631348,
"evidence_ids": [],
"preview": "eep USB-C power connected throughout the reset.\n\n## Warranty\n\nThe warranty lasts two years. Keep ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-7",
"rank": 3,
"score": 0.6855702996253967,
"evidence_ids": [],
"preview": "ing days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-3",
"rank": 4,
"score": 0.6573668718338013,
"evidence_ids": [],
"preview": "ars. Keep your receipt. Accidental water damage is excluded. Contact the support team before ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-1",
"rank": 5,
"score": 0.16223493218421936,
"evidence_ids": [],
"preview": " pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-5",
"rank": 6,
"score": 0.10600543022155762,
"evidence_ids": [],
"preview": "Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-4",
"rank": 7,
"score": 0.10283465683460236,
"evidence_ids": [],
"preview": "am before returning a device.\n\n## Battery care\n\nStore the device in a cool, dry place. Charge it ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-6",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "teries at an approved collection point.\n\n## Delivery\n\nStandard delivery takes five working days. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-8",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "he parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long should I hold the reset button, and what power is required?",
"limit": 20,
"locations": [
{
"child_id": "child-0",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 0,
"end": 99
},
{
"child_id": "child-2",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 178,
"end": 275
},
{
"child_id": "child-7",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 610,
"end": 704
},
{
"child_id": "child-3",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 265,
"end": 358
},
{
"child_id": "child-1",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 89,
"end": 188
},
{
"child_id": "child-5",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 435,
"end": 533
},
{
"child_id": "child-4",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 348,
"end": 445
},
{
"child_id": "child-6",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 523,
"end": 620
},
{
"child_id": "child-8",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 694,
"end": 713
}
]
}
},
{
"name": "parent_expansion",
"elapsed_ms": 0.008194998372346163,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.315727949142456,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-0",
"matched_child_start": 0,
"matched_child_end": 99,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How long should I hold the reset button, and what power is required?"
}
},
{
"name": "retrieval",
"elapsed_ms": 0.14025600103195757,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.315727949142456,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-0",
"matched_child_start": 0,
"matched_child_end": 99,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How long should I hold the reset button, and what power is required?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.03715999991982244,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.315727949142456,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-0",
"matched_child_start": 0,
"matched_child_end": 99,
"returned_context": "full_parent"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 741,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
# Orion device manual
## Reset pairing
Hold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.
## Warranty
The warranty lasts two years. Keep your receipt. Accidental water damage is excluded. Contact the support team before returning a device.
## Battery care
Store the device in a cool, dry place. Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.
## Delivery
Standard delivery takes five working days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel arrives.
Observed stages and selection evidence
[
{
"name": "children",
"elapsed_ms": 0.08000199886737391,
"hits": [
{
"source_unit_id": "child-3",
"rank": 1,
"score": 2.69175124168396,
"evidence_ids": [],
"preview": "ars. Keep your receipt. Accidental water damage is excluded. Contact the support team before ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-2",
"rank": 2,
"score": 1.2320934534072876,
"evidence_ids": [],
"preview": "eep USB-C power connected throughout the reset.\n\n## Warranty\n\nThe warranty lasts two years. Keep ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-7",
"rank": 3,
"score": 1.2126452922821045,
"evidence_ids": [],
"preview": "ing days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-1",
"rank": 4,
"score": 0.16223493218421936,
"evidence_ids": [],
"preview": " pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-0",
"rank": 5,
"score": 0.10937795042991638,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-5",
"rank": 6,
"score": 0.10600543022155762,
"evidence_ids": [],
"preview": "Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-4",
"rank": 7,
"score": 0.10283465683460236,
"evidence_ids": [],
"preview": "am before returning a device.\n\n## Battery care\n\nStore the device in a cool, dry place. Charge it ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-6",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "teries at an approved collection point.\n\n## Delivery\n\nStandard delivery takes five working days. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-8",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "he parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long is the warranty and what damage is excluded?",
"limit": 20,
"locations": [
{
"child_id": "child-3",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 265,
"end": 358
},
{
"child_id": "child-2",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 178,
"end": 275
},
{
"child_id": "child-7",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 610,
"end": 704
},
{
"child_id": "child-1",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 89,
"end": 188
},
{
"child_id": "child-0",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 0,
"end": 99
},
{
"child_id": "child-5",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 435,
"end": 533
},
{
"child_id": "child-4",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 348,
"end": 445
},
{
"child_id": "child-6",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 523,
"end": 620
},
{
"child_id": "child-8",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 694,
"end": 713
}
]
}
},
{
"name": "parent_expansion",
"elapsed_ms": 0.005971996870357543,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.69175124168396,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-3",
"matched_child_start": 265,
"matched_child_end": 358,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How long is the warranty and what damage is excluded?"
}
},
{
"name": "retrieval",
"elapsed_ms": 0.0980160002654884,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.69175124168396,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-3",
"matched_child_start": 265,
"matched_child_end": 358,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How long is the warranty and what damage is excluded?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.02166199919884093,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.69175124168396,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-3",
"matched_child_start": 265,
"matched_child_end": 358,
"returned_context": "full_parent"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 741,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
# Orion device manual
## Reset pairing
Hold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.
## Warranty
The warranty lasts two years. Keep your receipt. Accidental water damage is excluded. Contact the support team before returning a device.
## Battery care
Store the device in a cool, dry place. Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.
## Delivery
Standard delivery takes five working days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel arrives.
Observed stages and selection evidence
[
{
"name": "children",
"elapsed_ms": 0.07028399704722688,
"hits": [
{
"source_unit_id": "child-4",
"rank": 1,
"score": 1.2256488800048828,
"evidence_ids": [],
"preview": "am before returning a device.\n\n## Battery care\n\nStore the device in a cool, dry place. Charge it ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-5",
"rank": 2,
"score": 1.209876298904419,
"evidence_ids": [],
"preview": "Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-6",
"rank": 3,
"score": 0.7703071236610413,
"evidence_ids": [],
"preview": "teries at an approved collection point.\n\n## Delivery\n\nStandard delivery takes five working days. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-0",
"rank": 4,
"score": 0.5270749926567078,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-1",
"rank": 5,
"score": 0.0,
"evidence_ids": [],
"preview": " pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-2",
"rank": 6,
"score": 0.0,
"evidence_ids": [],
"preview": "eep USB-C power connected throughout the reset.\n\n## Warranty\n\nThe warranty lasts two years. Keep ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-3",
"rank": 7,
"score": 0.0,
"evidence_ids": [],
"preview": "ars. Keep your receipt. Accidental water damage is excluded. Contact the support team before ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-7",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "ing days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-8",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "he parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How often should I charge an unused device?",
"limit": 20,
"locations": [
{
"child_id": "child-4",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 348,
"end": 445
},
{
"child_id": "child-5",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 435,
"end": 533
},
{
"child_id": "child-6",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 523,
"end": 620
},
{
"child_id": "child-0",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 0,
"end": 99
},
{
"child_id": "child-1",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 89,
"end": 188
},
{
"child_id": "child-2",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 178,
"end": 275
},
{
"child_id": "child-3",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 265,
"end": 358
},
{
"child_id": "child-7",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 610,
"end": 704
},
{
"child_id": "child-8",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 694,
"end": 713
}
]
}
},
{
"name": "parent_expansion",
"elapsed_ms": 0.004427998646860942,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 1.2256488800048828,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-4",
"matched_child_start": 348,
"matched_child_end": 445,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How often should I charge an unused device?"
}
},
{
"name": "retrieval",
"elapsed_ms": 0.08442000034847297,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 1.2256488800048828,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-4",
"matched_child_start": 348,
"matched_child_end": 445,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "How often should I charge an unused device?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.019537001207936555,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 1.2256488800048828,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-4",
"matched_child_start": 348,
"matched_child_end": 445,
"returned_context": "full_parent"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 741,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
# Orion device manual
## Reset pairing
Hold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.
## Warranty
The warranty lasts two years. Keep your receipt. Accidental water damage is excluded. Contact the support team before returning a device.
## Battery care
Store the device in a cool, dry place. Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.
## Delivery
Standard delivery takes five working days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel arrives.
Observed stages and selection evidence
[
{
"name": "children",
"elapsed_ms": 0.07966200064402074,
"hits": [
{
"source_unit_id": "child-7",
"rank": 1,
"score": 2.4967360496520996,
"evidence_ids": [],
"preview": "ing days. Tracked delivery is available in Morocco. Keep the tracking number until the parcel ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-6",
"rank": 2,
"score": 0.8006759881973267,
"evidence_ids": [],
"preview": "teries at an approved collection point.\n\n## Delivery\n\nStandard delivery takes five working days. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-3",
"rank": 3,
"score": 0.5443947315216064,
"evidence_ids": [],
"preview": "ars. Keep your receipt. Accidental water damage is excluded. Contact the support team before ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-0",
"rank": 4,
"score": 0.0,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-1",
"rank": 5,
"score": 0.0,
"evidence_ids": [],
"preview": " pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-2",
"rank": 6,
"score": 0.0,
"evidence_ids": [],
"preview": "eep USB-C power connected throughout the reset.\n\n## Warranty\n\nThe warranty lasts two years. Keep ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-4",
"rank": 7,
"score": 0.0,
"evidence_ids": [],
"preview": "am before returning a device.\n\n## Battery care\n\nStore the device in a cool, dry place. Charge it ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-5",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "Charge it every three months when unused. Use only the supplied USB-C cable. Recycle batteries at ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "child-8",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "he parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "Where is tracked delivery available?",
"limit": 20,
"locations": [
{
"child_id": "child-7",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 610,
"end": 704
},
{
"child_id": "child-6",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 523,
"end": 620
},
{
"child_id": "child-3",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 265,
"end": 358
},
{
"child_id": "child-0",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 0,
"end": 99
},
{
"child_id": "child-1",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 89,
"end": 188
},
{
"child_id": "child-2",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 178,
"end": 275
},
{
"child_id": "child-4",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 348,
"end": 445
},
{
"child_id": "child-5",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 435,
"end": 533
},
{
"child_id": "child-8",
"parent_id": "doc-aaa8b734a101d7f34afb",
"start": 694,
"end": 713
}
]
}
},
{
"name": "parent_expansion",
"elapsed_ms": 0.005029000021750107,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.4967360496520996,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-7",
"matched_child_start": 610,
"matched_child_end": 704,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "Where is tracked delivery available?"
}
},
{
"name": "retrieval",
"elapsed_ms": 0.09733499973663129,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.4967360496520996,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-7",
"matched_child_start": 610,
"matched_child_end": 704,
"returned_context": "full_parent"
}
}
],
"details": {
"query": "Where is tracked delivery available?",
"limit": 1,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.02251300247735344,
"hits": [
{
"source_unit_id": "doc-aaa8b734a101d7f34afb",
"rank": 1,
"score": 2.4967360496520996,
"evidence_ids": [],
"preview": "# Orion device manual\n\n## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected throughout the reset.\n\n## Warranty\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"matched_child_id": "child-7",
"matched_child_start": 610,
"matched_child_end": 704,
"returned_context": "full_parent"
}
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 741,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
## Reset pairing
Rank 1 · manual.md · Source text · Original characters 41–98
Hold the reset button for eight seconds to clear pairing.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.11848500071209855,
"hits": [
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 1,
"score": 2.2885582447052,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 2,
"score": 0.9809407591819763,
"evidence_ids": [],
"preview": "connected throughout the reset.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 3,
"score": 0.8127185106277466,
"evidence_ids": [],
"preview": "tus light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 4,
"score": 0.6282253265380859,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 5,
"score": 0.5550644397735596,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 6,
"score": 0.15764456987380981,
"evidence_ids": [],
"preview": " Morocco. Keep the tracking number until the parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 7,
"score": 0.11353020370006561,
"evidence_ids": [],
"preview": "excluded. Contact the support team before returning a device.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 8,
"score": 0.08988720178604126,
"evidence_ids": [],
"preview": "n unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 9,
"score": 0.08405247330665588,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-c341c5c87a667038141c",
"rank": 10,
"score": 0.0,
"evidence_ids": [],
"preview": "# Orion device manual\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long should I hold the reset button, and what power is required?",
"limit": 20,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "compression",
"elapsed_ms": 0.3726879986061249,
"hits": [
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 1,
"score": 2.2885582447052,
"evidence_ids": [],
"preview": "## Reset pairing\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 0,
"end": 18
},
{
"start": 18,
"end": 75
}
]
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 2,
"score": 0.9809407591819763,
"evidence_ids": [],
"preview": "connected throughout the reset.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 2
},
"context_spans": [
{
"start": 0,
"end": 31
}
]
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 3,
"score": 0.8127185106277466,
"evidence_ids": [],
"preview": " Disconnect the companion app before starting.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 3
},
"context_spans": [
{
"start": 24,
"end": 70
},
{
"start": 70,
"end": 98
}
]
},
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 4,
"score": 0.6282253265380859,
"evidence_ids": [],
"preview": "The warranty lasts two years.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 4
},
"context_spans": [
{
"start": 13,
"end": 42
},
{
"start": 61,
"end": 98
}
]
},
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 5,
"score": 0.5550644397735596,
"evidence_ids": [],
"preview": " Tracked delivery is available in Morocco.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 5
},
"context_spans": [
{
"start": 55,
"end": 97
}
]
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 6,
"score": 0.15764456987380981,
"evidence_ids": [],
"preview": " Keep the tracking number until the parcel arrives.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 6
},
"context_spans": [
{
"start": 9,
"end": 60
}
]
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 7,
"score": 0.11353020370006561,
"evidence_ids": [],
"preview": " Contact the support team before returning a device.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 7
},
"context_spans": [
{
"start": 9,
"end": 61
}
]
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 8,
"score": 0.08988720178604126,
"evidence_ids": [],
"preview": " Use only the supplied USB-C cable.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 8
},
"context_spans": [
{
"start": 9,
"end": 44
}
]
},
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 9,
"score": 0.08405247330665588,
"evidence_ids": [],
"preview": "Store the device in a cool, dry place.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 9
},
"context_spans": [
{
"start": 17,
"end": 55
}
]
}
],
"details": {
"mode": "sentences",
"max_spans_per_source": 2,
"input_ids": [
"passage-5bdc4916bcb2462d0250",
"passage-a9bffaf7e82ff6d02b1e",
"passage-fc31aa14f18cf8b2a89a",
"passage-4be5e239b1ab47982a8e",
"passage-1518103967d65de40d95",
"passage-3657f06cb41040af19b1",
"passage-6c50fe60990136f40ac3",
"passage-a73c9da59294e4af482b",
"passage-32a3368b3b75076e202c",
"passage-c341c5c87a667038141c"
],
"order": "candidate rank, then original character offset",
"budget_policy": "final context selection keeps whole compressed source groups",
"failure_policy": "fail",
"state": "complete",
"selections": {
"passage-5bdc4916bcb2462d0250": [
{
"start": 0,
"end": 18
},
{
"start": 18,
"end": 75
}
],
"passage-a9bffaf7e82ff6d02b1e": [
{
"start": 0,
"end": 31
}
],
"passage-fc31aa14f18cf8b2a89a": [
{
"start": 24,
"end": 70
},
{
"start": 70,
"end": 98
}
],
"passage-4be5e239b1ab47982a8e": [
{
"start": 13,
"end": 42
},
{
"start": 61,
"end": 98
}
],
"passage-1518103967d65de40d95": [
{
"start": 55,
"end": 97
}
],
"passage-3657f06cb41040af19b1": [
{
"start": 9,
"end": 60
}
],
"passage-6c50fe60990136f40ac3": [
{
"start": 9,
"end": 61
}
],
"passage-a73c9da59294e4af482b": [
{
"start": 9,
"end": 44
}
],
"passage-32a3368b3b75076e202c": [
{
"start": 17,
"end": 55
}
],
"passage-c341c5c87a667038141c": []
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.03204099994036369,
"hits": [
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 1,
"score": 2.2885582447052,
"evidence_ids": [],
"preview": "## Reset pairing\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 0,
"end": 18
},
{
"start": 18,
"end": 75
}
]
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 211,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
The warranty lasts two years.
Rank 1 · manual.md · Source text · Original characters 288–325
Accidental water damage is excluded.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.09342700286651962,
"hits": [
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 1,
"score": 3.4753642082214355,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 2,
"score": 1.1101288795471191,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 3,
"score": 0.7659269571304321,
"evidence_ids": [],
"preview": "excluded. Contact the support team before returning a device.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 4,
"score": 0.15764456987380981,
"evidence_ids": [],
"preview": " Morocco. Keep the tracking number until the parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 5,
"score": 0.14540082216262817,
"evidence_ids": [],
"preview": "connected throughout the reset.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 6,
"score": 0.13330140709877014,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 7,
"score": 0.09311925619840622,
"evidence_ids": [],
"preview": "tus light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 8,
"score": 0.08988720178604126,
"evidence_ids": [],
"preview": "n unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 9,
"score": 0.08405247330665588,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-c341c5c87a667038141c",
"rank": 10,
"score": 0.0,
"evidence_ids": [],
"preview": "# Orion device manual\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How long is the warranty and what damage is excluded?",
"limit": 20,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "compression",
"elapsed_ms": 0.24530600057914853,
"hits": [
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 1,
"score": 3.4753642082214355,
"evidence_ids": [],
"preview": "The warranty lasts two years.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 13,
"end": 42
},
{
"start": 61,
"end": 98
}
]
},
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 2,
"score": 1.1101288795471191,
"evidence_ids": [],
"preview": " Tracked delivery is available in Morocco.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 2
},
"context_spans": [
{
"start": 55,
"end": 97
}
]
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 3,
"score": 0.7659269571304321,
"evidence_ids": [],
"preview": "excluded.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 3
},
"context_spans": [
{
"start": 0,
"end": 9
},
{
"start": 9,
"end": 61
}
]
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 4,
"score": 0.15764456987380981,
"evidence_ids": [],
"preview": " Keep the tracking number until the parcel arrives.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 4
},
"context_spans": [
{
"start": 9,
"end": 60
}
]
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 5,
"score": 0.14540082216262817,
"evidence_ids": [],
"preview": "connected throughout the reset.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 5
},
"context_spans": [
{
"start": 0,
"end": 31
}
]
},
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 6,
"score": 0.13330140709877014,
"evidence_ids": [],
"preview": "Hold the reset button for eight seconds to clear pairing.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 6
},
"context_spans": [
{
"start": 18,
"end": 75
},
{
"start": 75,
"end": 93
}
]
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 7,
"score": 0.09311925619840622,
"evidence_ids": [],
"preview": " Disconnect the companion app before starting.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 7
},
"context_spans": [
{
"start": 24,
"end": 70
}
]
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 8,
"score": 0.08988720178604126,
"evidence_ids": [],
"preview": " Use only the supplied USB-C cable.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 8
},
"context_spans": [
{
"start": 9,
"end": 44
}
]
},
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 9,
"score": 0.08405247330665588,
"evidence_ids": [],
"preview": "Store the device in a cool, dry place.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 9
},
"context_spans": [
{
"start": 17,
"end": 55
}
]
}
],
"details": {
"mode": "sentences",
"max_spans_per_source": 2,
"input_ids": [
"passage-4be5e239b1ab47982a8e",
"passage-1518103967d65de40d95",
"passage-6c50fe60990136f40ac3",
"passage-3657f06cb41040af19b1",
"passage-a9bffaf7e82ff6d02b1e",
"passage-5bdc4916bcb2462d0250",
"passage-fc31aa14f18cf8b2a89a",
"passage-a73c9da59294e4af482b",
"passage-32a3368b3b75076e202c",
"passage-c341c5c87a667038141c"
],
"order": "candidate rank, then original character offset",
"budget_policy": "final context selection keeps whole compressed source groups",
"failure_policy": "fail",
"state": "complete",
"selections": {
"passage-4be5e239b1ab47982a8e": [
{
"start": 13,
"end": 42
},
{
"start": 61,
"end": 98
}
],
"passage-1518103967d65de40d95": [
{
"start": 55,
"end": 97
}
],
"passage-6c50fe60990136f40ac3": [
{
"start": 0,
"end": 9
},
{
"start": 9,
"end": 61
}
],
"passage-3657f06cb41040af19b1": [
{
"start": 9,
"end": 60
}
],
"passage-a9bffaf7e82ff6d02b1e": [
{
"start": 0,
"end": 31
}
],
"passage-5bdc4916bcb2462d0250": [
{
"start": 18,
"end": 75
},
{
"start": 75,
"end": 93
}
],
"passage-fc31aa14f18cf8b2a89a": [
{
"start": 24,
"end": 70
}
],
"passage-a73c9da59294e4af482b": [
{
"start": 9,
"end": 44
}
],
"passage-32a3368b3b75076e202c": [
{
"start": 17,
"end": 55
}
],
"passage-c341c5c87a667038141c": []
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.026279998564859852,
"hits": [
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 1,
"score": 3.4753642082214355,
"evidence_ids": [],
"preview": "The warranty lasts two years.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 13,
"end": 42
},
{
"start": 61,
"end": 98
}
]
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 192,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
Store the device in a cool, dry place.
Rank 1 · manual.md · Source text · Original characters 434–476
Charge it every three months when unused.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.08580299981986172,
"hits": [
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 1,
"score": 1.5058517456054688,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 2,
"score": 1.2111560106277466,
"evidence_ids": [],
"preview": "n unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-c341c5c87a667038141c",
"rank": 3,
"score": 0.6842033267021179,
"evidence_ids": [],
"preview": "# Orion device manual\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 4,
"score": 0.5042375326156616,
"evidence_ids": [],
"preview": "excluded. Contact the support team before returning a device.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 5,
"score": 0.0,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 6,
"score": 0.0,
"evidence_ids": [],
"preview": "tus light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 7,
"score": 0.0,
"evidence_ids": [],
"preview": "connected throughout the reset.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 10,
"score": 0.0,
"evidence_ids": [],
"preview": " Morocco. Keep the tracking number until the parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "How often should I charge an unused device?",
"limit": 20,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "compression",
"elapsed_ms": 0.1759549995767884,
"hits": [
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 1,
"score": 1.5058517456054688,
"evidence_ids": [],
"preview": "Store the device in a cool, dry place.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 17,
"end": 55
},
{
"start": 55,
"end": 97
}
]
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 2,
"score": 1.2111560106277466,
"evidence_ids": [],
"preview": "n unused.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 2
},
"context_spans": [
{
"start": 0,
"end": 9
},
{
"start": 44,
"end": 95
}
]
},
{
"source_unit_id": "passage-c341c5c87a667038141c",
"rank": 3,
"score": 0.6842033267021179,
"evidence_ids": [],
"preview": "# Orion device manual\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 3
},
"context_spans": [
{
"start": 0,
"end": 23
}
]
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 4,
"score": 0.5042375326156616,
"evidence_ids": [],
"preview": " Contact the support team before returning a device.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 4
},
"context_spans": [
{
"start": 9,
"end": 61
}
]
}
],
"details": {
"mode": "sentences",
"max_spans_per_source": 2,
"input_ids": [
"passage-32a3368b3b75076e202c",
"passage-a73c9da59294e4af482b",
"passage-c341c5c87a667038141c",
"passage-6c50fe60990136f40ac3",
"passage-5bdc4916bcb2462d0250",
"passage-fc31aa14f18cf8b2a89a",
"passage-a9bffaf7e82ff6d02b1e",
"passage-4be5e239b1ab47982a8e",
"passage-1518103967d65de40d95",
"passage-3657f06cb41040af19b1"
],
"order": "candidate rank, then original character offset",
"budget_policy": "final context selection keeps whole compressed source groups",
"failure_policy": "fail",
"state": "complete",
"selections": {
"passage-32a3368b3b75076e202c": [
{
"start": 17,
"end": 55
},
{
"start": 55,
"end": 97
}
],
"passage-a73c9da59294e4af482b": [
{
"start": 0,
"end": 9
},
{
"start": 44,
"end": 95
}
],
"passage-c341c5c87a667038141c": [
{
"start": 0,
"end": 23
}
],
"passage-6c50fe60990136f40ac3": [
{
"start": 9,
"end": 61
}
],
"passage-5bdc4916bcb2462d0250": [],
"passage-fc31aa14f18cf8b2a89a": [],
"passage-a9bffaf7e82ff6d02b1e": [],
"passage-4be5e239b1ab47982a8e": [],
"passage-1518103967d65de40d95": [],
"passage-3657f06cb41040af19b1": []
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.02223199771833606,
"hits": [
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 1,
"score": 1.5058517456054688,
"evidence_ids": [],
"preview": "Store the device in a cool, dry place.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 17,
"end": 55
},
{
"start": 55,
"end": 97
}
]
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 214,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
## Delivery
Rank 1 · manual.md · Source text · Original characters 619–661
Tracked delivery is available in Morocco.
Observed stages and selection evidence
[
{
"name": "retrieval",
"elapsed_ms": 0.130978998640785,
"hits": [
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 1,
"score": 3.3280820846557617,
"evidence_ids": [],
"preview": "## Delivery\n\nStandard delivery takes five working days. Tracked delivery is available in Morocco. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 2,
"score": 0.5351060628890991,
"evidence_ids": [],
"preview": "## Warranty\n\nThe warranty lasts two years. Keep your receipt. Accidental water damage is excluded. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-c341c5c87a667038141c",
"rank": 3,
"score": 0.0,
"evidence_ids": [],
"preview": "# Orion device manual\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-5bdc4916bcb2462d0250",
"rank": 4,
"score": 0.0,
"evidence_ids": [],
"preview": "## Reset pairing\n\nHold the reset button for eight seconds to clear pairing. The status light ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-fc31aa14f18cf8b2a89a",
"rank": 5,
"score": 0.0,
"evidence_ids": [],
"preview": "tus light flashes twice. Disconnect the companion app before starting. Keep USB-C power connected ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a9bffaf7e82ff6d02b1e",
"rank": 6,
"score": 0.0,
"evidence_ids": [],
"preview": "connected throughout the reset.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-6c50fe60990136f40ac3",
"rank": 7,
"score": 0.0,
"evidence_ids": [],
"preview": "excluded. Contact the support team before returning a device.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-32a3368b3b75076e202c",
"rank": 8,
"score": 0.0,
"evidence_ids": [],
"preview": "## Battery care\n\nStore the device in a cool, dry place. Charge it every three months when unused. ",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-a73c9da59294e4af482b",
"rank": 9,
"score": 0.0,
"evidence_ids": [],
"preview": "n unused. Use only the supplied USB-C cable. Recycle batteries at an approved collection point.\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
},
{
"source_unit_id": "passage-3657f06cb41040af19b1",
"rank": 10,
"score": 0.0,
"evidence_ids": [],
"preview": " Morocco. Keep the tracking number until the parcel arrives.\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb"
}
}
],
"details": {
"query": "Where is tracked delivery available?",
"limit": 20,
"index_cache": "not_applicable",
"model_cache": "unknown",
"runtime_versions": {
"bm25s": "0.2.14",
"numpy": "2.5.4"
}
}
},
{
"name": "compression",
"elapsed_ms": 0.16995300029520877,
"hits": [
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 1,
"score": 3.3280820846557617,
"evidence_ids": [],
"preview": "## Delivery\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 0,
"end": 13
},
{
"start": 55,
"end": 97
}
]
},
{
"source_unit_id": "passage-4be5e239b1ab47982a8e",
"rank": 2,
"score": 0.5351060628890991,
"evidence_ids": [],
"preview": " Accidental water damage is excluded.",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 2
},
"context_spans": [
{
"start": 61,
"end": 98
}
]
}
],
"details": {
"mode": "sentences",
"max_spans_per_source": 2,
"input_ids": [
"passage-1518103967d65de40d95",
"passage-4be5e239b1ab47982a8e",
"passage-c341c5c87a667038141c",
"passage-5bdc4916bcb2462d0250",
"passage-fc31aa14f18cf8b2a89a",
"passage-a9bffaf7e82ff6d02b1e",
"passage-6c50fe60990136f40ac3",
"passage-32a3368b3b75076e202c",
"passage-a73c9da59294e4af482b",
"passage-3657f06cb41040af19b1"
],
"order": "candidate rank, then original character offset",
"budget_policy": "final context selection keeps whole compressed source groups",
"failure_policy": "fail",
"state": "complete",
"selections": {
"passage-1518103967d65de40d95": [
{
"start": 0,
"end": 13
},
{
"start": 55,
"end": 97
}
],
"passage-4be5e239b1ab47982a8e": [
{
"start": 61,
"end": 98
}
],
"passage-c341c5c87a667038141c": [],
"passage-5bdc4916bcb2462d0250": [],
"passage-fc31aa14f18cf8b2a89a": [],
"passage-a9bffaf7e82ff6d02b1e": [],
"passage-6c50fe60990136f40ac3": [],
"passage-32a3368b3b75076e202c": [],
"passage-a73c9da59294e4af482b": [],
"passage-3657f06cb41040af19b1": []
}
}
},
{
"name": "context_selection",
"elapsed_ms": 0.02344399763387628,
"hits": [
{
"source_unit_id": "passage-1518103967d65de40d95",
"rank": 1,
"score": 3.3280820846557617,
"evidence_ids": [],
"preview": "## Delivery\n\n",
"metadata": {
"document_id": "doc-aaa8b734a101d7f34afb",
"compression_input_rank": 1
},
"context_spans": [
{
"start": 0,
"end": 13
},
{
"start": 55,
"end": 97
}
]
}
],
"details": {
"adapter": "unicode-codepoints-v1",
"unit": "characters",
"exact": true,
"count": 181,
"budget": null,
"scope": "rendered_retrieval_context",
"skipped": [],
"policy": "ranked-whole-passages-v1"
}
}
]
{
"splits": {
"dev": {
"all": {
"answerable_questions": 4,
"changes": {
"improved": 1,
"regressed": 0,
"stable_pass": 3,
"stable_fail": 0
},
"intervals": {
"sufficiency_rate": {
"estimate": 0.25,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.75,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_evidence_coverage": {
"estimate": 0.125,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.375,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_reciprocal_rank": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_precision_at_k": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
}
}
}
},
"heldout": {
"all": {
"answerable_questions": 0,
"changes": {
"improved": 0,
"regressed": 0,
"stable_pass": 0,
"stable_fail": 0
},
"intervals": {
"sufficiency_rate": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_evidence_coverage": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_reciprocal_rank": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_precision_at_k": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
}
}
}
}
},
"questions": [
{
"question_id": "Q1",
"split": "dev",
"tags": [],
"answerable": true,
"change": "improved",
"gained": [
"POWER"
],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 1.0,
"mean_evidence_coverage": 0.5,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q2",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q3",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q4",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
}
]
}
## compress versus bm25
Observed gains/losses under the declared policy; no automatic winner
{
"splits": {
"dev": {
"all": {
"answerable_questions": 4,
"changes": {
"improved": 0,
"regressed": 0,
"stable_pass": 3,
"stable_fail": 1
},
"intervals": {
"sufficiency_rate": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_evidence_coverage": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_reciprocal_rank": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
},
"mean_precision_at_k": {
"estimate": 0.0,
"question_count": 4,
"group_count": 4,
"lower": 0.0,
"upper": 0.0,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "Small samples and reused public heldout data limit inference"
}
}
}
},
"heldout": {
"all": {
"answerable_questions": 0,
"changes": {
"improved": 0,
"regressed": 0,
"stable_pass": 0,
"stable_fail": 0
},
"intervals": {
"sufficiency_rate": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_evidence_coverage": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_reciprocal_rank": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
},
"mean_precision_at_k": {
"estimate": null,
"question_count": 0,
"group_count": 0,
"lower": null,
"upper": null,
"seed": 42,
"samples": 2000,
"method": "paired-cluster-percentile-bootstrap-v1",
"numpy_version": "2.5.4",
"confidence": 0.95,
"note": "No interval: fewer than two independent declared groups"
}
}
}
}
},
"questions": [
{
"question_id": "Q1",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_fail",
"gained": [],
"lost": [],
"missing": [
"POWER"
],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q2",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q3",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
},
{
"question_id": "Q4",
"split": "dev",
"tags": [],
"answerable": true,
"change": "stable_pass",
"gained": [],
"lost": [],
"missing": [],
"deltas": {
"sufficiency_rate": 0.0,
"mean_evidence_coverage": 0.0,
"mean_reciprocal_rank": 0.0,
"mean_precision_at_k": 0.0
}
}
]
}
## Limitations
- Grids are exploratory. Intervals are not adjusted for multiple comparisons.
- Heldout reuse is tracked under this output root, without protection against edits.
- Vector reuse is observed. Model/OS cache warmth and external traces are unknown.
- Missing timing/cost is unavailable. Supplied usage/pricing yields labeled estimates.
- nDCG requires graded relevance gains. Answer quality is not evaluated.