Supported files and schemas
TXT, Markdown, HTML, document JSON/JSONL, CSV and optional text-bearing PDF.
On this page
Supported input formats
| Format | Searchable text and retained locations |
|---|---|
| UTF-8 TXT / Markdown | Original character and line offsets; Markdown ATX heading paths outside fenced code. |
| HTML / HTM | Visible text grouped by headings, section number and heading path. No scripts or remote assets execute. |
| JSON / JSONL | Document object(s), one selected text field per record; record and file line where available. |
| CSV | Unique header and consistent comma-separated rows; data-row and physical-line ranges, including multiline cells. |
| PDF + pdf extra | Text-bearing pages and original page numbers; parser/version/warnings recorded. |
Document records
[{"text":"Hold reset for eight seconds.","product":"Orion","version":2}]Select one unambiguous text, content or body string field; --text-field NAME resolves multiple candidates or custom names. Nested metadata is retained without being flattened into text. Question/evaluation records and RAGvidence artifacts are rejected as documents. CSV metadata remains string-typed.
ragvi bm25 ./articles.json --text-field article --query "Reset procedure"Folder discovery and limits
Discovery is recursive and deterministic. Hidden files, symlinks, known generated directories, selected questions/config/env files, output/cache roots and marked earlier runs are excluded. Reports list intentionally skipped files, extraction failures and parser warnings.
Defaults: 1,000 files; 20 MiB per file; 100 MiB attempted bytes; 10,000 records/sections/pages per file; 20 Mi extracted characters. --max-files, --max-file-bytes, --max-total-bytes, --max-records and --max-extracted-characters override these limits. Total budgets fail the command; valid files can proceed past individual folder extraction failures.
ragvi bm25 ./data --include '*.jsonl' --include '*.csv' --exclude 'drafts/*' --no-inputExtraction limits
No OCR/scanned PDF, DOCX/spreadsheet, image/audio, crawling or database connector support. PDF/HTML extraction does not guarantee table/layout fidelity. Encrypted and fully textless PDFs fail. CSS layout/dynamic HTML and Setext headings are not interpreted. Unicode decoding is not language-specific stemming or universal semantic model support.