RAG Store Guide

A context store is a collection of documents you upload. When you attach a store to a comparison, evaluation, or benchmark, LLM Prover retrieves the most relevant chunks from your documents and injects them into the prompt automatically. The model answers from your content – not from its training data.

This matters for two reasons. First, models have a knowledge cutoff and no access to your internal documentation, product data, or proprietary content. A context store closes that gap. Second, it changes what you are testing: without a store you are testing the model’s general knowledge and instruction-following; with one you are testing retrieval quality and how well the model uses provided context. Both are valid tests. They are different tests.

This guide covers how to create a store, what affects retrieval quality, how to diagnose retrieval failures, and how stores interact with scoring.

What a context store is and when to use one

Ground model outputs in your data

When your application uses internal documentation, product specs, support articles, or any content the model was not trained on, a context store makes that content available at run time. The model answers from what you uploaded, not from what it knows.

Test your RAG pipeline

If your production system uses retrieval-augmented generation, testing without a store tells you nothing about production behaviour. Attach your actual knowledge base to get results that predict how your system will perform.

Evaluate faithfulness

With a context store attached, you can score whether the model’s response is faithful to the retrieved content – whether it stays within what the documents say rather than hallucinating beyond them. This is a distinct scoring dimension from general quality.

Creating a store and uploading documents

1. Navigate to Stores in the sidebar (Config section)
2. Click New Store
3. Give it a name that identifies its content – “Product docs v3”, “Support KB”, “Onboarding materials”
4. Upload your documents (see supported formats below)
5. Wait for ingest – status changes from pending to ready. For large documents this may take up to 30 seconds. Click refresh if needed.

Once status is ready, the store appears in the store picker on any comparison, evaluation, or benchmark form.

Supported formats: PDF, TXT, MD, DOCX, CSV, JSON, XLSX

Tier limits:

• Pro: 1 store, up to 20 files, 100MB total, 5MB per file
• Pro+: 5 stores, up to 100 files, 1GB total, 10MB per file
• Enterprise: unlimited stores and storage, 25MB per file, all file types

Chunking quality and why it matters

When you upload a document, LLM Prover splits it into chunks and embeds them for retrieval. When a run fires, the most relevant chunks are retrieved based on semantic similarity to the prompt. The quality of retrieval depends entirely on how well the document was chunked.

Structured chunking (DOCX, Markdown)

DOCX and Markdown files are chunked by heading structure – each heading and its content becomes a chunk. Section boundaries are preserved. A 30-section document produces 30 retrievable chunks, each containing exactly the content under that heading.

Best-effort chunking (PDF, TXT, CSV, XLSX)

PDF and plain text files are chunked by paragraph or token boundary. Section headings may be split from their content. A 30-section PDF may produce 8 chunks of mixed content, making it hard for retrieval to find the right section.

The practical implication: a file that produces 2 chunks instead of 20 means most of the document is unretrievable. The model answers from whatever was in those 2 chunks, regardless of whether they contain the relevant information.

Chunking quality indicator

Each file in your store shows a chunking quality indicator: structured (DOCX, MD) or best-effort (PDF, TXT, CSV). If a file shows best-effort, consider re-uploading in a structured format before relying on it for evaluations.

Document structure for reliable retrieval

Even with DOCX or Markdown, document structure affects retrieval quality. A well-structured document retrieves accurately. A poorly structured one does not.

Use clear headings

Each major topic should have its own heading. The heading text is part of the chunk and helps retrieval match the right section to the prompt. “Section 3.2” is a poor heading. “Data retention policy” is a good one.

One topic per section

If a section covers multiple topics, retrieval may return it for the wrong query. Split mixed sections into separate headed sections.

Separate examples from rules

This is the most common retrieval problem in compliance documents. If a section contains both a rule and an example of violating it, retrieval may return the example when the rule is what you need – or vice versa.

Avoid tables for prose content

Tables in DOCX are chunked as a single block. If your document uses tables to present policy rules, each row is not independently retrievable. Convert policy tables to headed sections where possible.

Terminology and retrieval matching

Retrieval works by semantic similarity – it finds chunks whose meaning is closest to the prompt. Terminology mismatches reduce retrieval accuracy even when the right content exists in the document.

Match your prompt language to your document language

If your document says “personally identifiable information” and your prompt says “personal data”, retrieval may not find the right section. Use the same terms in your prompts and criteria as appear in your documents.

Acronyms and abbreviations

If your document uses “GDPR” throughout and your prompt says “General Data Protection Regulation”, retrieval may miss matches. Prefer the form used in the document.

Version-specific terminology

If your document refers to “v2 API” and your prompt says “new API”, retrieval will not reliably connect them. Be specific.

Attaching a store to a run

Comparisons and evaluations

On the comparison or evaluation form, select your store from the Store dropdown. The store name appears in the result config box. Retrieved chunks are visible in the result drawer – check them to confirm the right content was retrieved.

Benchmarks

Select a store when creating or editing a benchmark. The store is saved with the benchmark configuration and used on every scheduled run. If you update the store (re-upload a document), the next run uses the updated content automatically.

What the model receives

The model receives the system prompt (if attached), the retrieved chunks, and the user prompt – in that order. The number of chunks injected is controlled by the rag_context_tokens limit for your tier. Starter: 500 tokens. Pro and above: higher limits apply.

Stores and scoring

Attaching a store changes what you are testing and therefore what your scores mean.

Without a store: scores reflect the model’s general knowledge and instruction-following on your prompt.

With a store: scores reflect how well the model uses the retrieved content. A high score means the response is consistent with what was retrieved. A low score may mean the model ignored the context, the wrong chunks were retrieved, or the prompt and document are misaligned.

Faithfulness scoring

When a store is attached, the heuristic scorer includes a faithfulness dimension – it measures whether the response stays within the retrieved content rather than introducing content not in the chunks. This is separate from accuracy. A response can be accurate (correct answer) but unfaithful (answer not grounded in the retrieved chunks).

Rubric scoring with a store

If you attach both a store and a rubric, the judge scores each criterion against the response in the context of the retrieved chunks. Write criteria that reference the context explicitly: “The response must only cite information present in the provided context. Claims not supported by the context must not be made.”

The diagnostic judge and retrieval

The diagnostic judge flags retrieval problems automatically. Common findings:

• Too few chunks retrieved – the document may need restructuring
• Terminology mismatch – prompt and document use different terms
• Mixed chunk – the retrieved chunk contains both relevant and irrelevant content
• Low confidence match – the retrieved chunk is a weak match for the prompt

Check the Diagnostics tab when scores are lower than expected on a RAG-powered run. Retrieval problems are the most common cause of unexpectedly low scores on well-written rubrics.

Multiple stores and store types

Context stores vs compliance stores

A context store grounds model responses in your knowledge base. A compliance store is used by the judge to score responses against your regulations – the model never sees it directly. These are separate store types with separate purposes. Do not mix them.

Multiple stores (Pro+ and above)

Pro+ supports up to 5 stores. Use separate stores for separate knowledge domains – product documentation, support articles, legal policies. Attach the relevant store to each run rather than combining all content into one store. Smaller, focused stores produce better retrieval than large mixed ones.

Updating a store

Re-upload a document to update its content. The new version is used on the next run. Previous run results are not affected – they reflect the store content at the time of the run.

Common mistakes

Uploading a PDF when DOCX is available

The most common cause of poor retrieval. If the document exists as a Word file or Google Doc, download it as DOCX. The retrieval improvement is significant.

One large file instead of structured sections

A single 50-page PDF produces far fewer usable chunks than the same content split into headed DOCX sections. Structure is retrieval.

Not checking retrieved chunks

The result drawer shows which chunks were retrieved. Check them before concluding that retrieval is working. A run that produces a plausible-looking response may have retrieved the wrong chunks entirely.

Testing without a store when production uses one

If your application uses RAG, test with RAG. A comparison without a store does not predict production behaviour.

Using the same store for context and compliance

Context stores and compliance stores serve different purposes and are queried differently. Keep them separate.

What’s next