Compliance Store Guide

A compliance store is a collection of regulation or policy documents you upload. When you attach one to an evaluation or benchmark, the judge retrieves the most relevant clause per criterion and scores the model response against that clause – not against the judge’s training knowledge of the regulation. The regulation you uploaded is the standard. Not what the model knows. Not what the judge knows.

Context stores vs compliance stores

This distinction matters before anything else.

• Context store: injected into the prompt. The model reads it and answers from it.
• Compliance store: never seen by the model. Used only by the judge, per criterion, after the response is generated.

Same upload interface, different store type selector. Do not mix them.

A compliance store attached to a run that has no rubric does nothing – it is only queried during rubric scoring.

When to use a compliance store

Score model outputs against your internal policies

HR policies, product usage terms, support response standards. The judge checks whether responses are consistent with what your documents actually say.

Regulatory consistency testing

GDPR, HIPAA, SOC 2, financial regulations. Upload the relevant sections and score whether outputs make claims that contradict them. This is consistency measurement, not certification.

Drift detection over time

Run the same benchmark weekly. If a model update or prompt change causes outputs to drift from your policy documents, the score drops. You see it before it reaches users.

Creating a compliance store

1. Navigate to Stores in the sidebar
2. Click New Store
3. Select store type: Compliance (not Context)
4. Name it to identify the regulation or policy – “GDPR Article 5-7”, “Support Response Policy v2”, “Product Terms of Service”
5. Upload documents
6. Wait for ingest – status changes to ready

Tier limits (compliance stores count against the same store quota as context stores):

• Pro+: up to 5 stores total (context + compliance combined), 100 files, 1GB total, 10MB per file
• Enterprise: unlimited stores and storage, 25MB per file

Document structure for reliable clause retrieval

Numbered clauses are the most retrievable structure

Regulations and policies with numbered clauses (1.1, 2.3.a) produce the best retrieval. Each clause becomes a distinct chunk. The judge retrieves the clause most relevant to the criterion and scores against it. “Article 5(1)(a)” is a better chunk boundary than “paragraph 3 of the data minimisation section.”

Use DOCX or Markdown

Same rule as context stores. PDF compliance documents are chunked by paragraph boundary, not by clause structure. A 40-clause regulation in PDF may produce 8 chunks of mixed clauses. Upload as DOCX or Markdown where possible.

Separate compliant and non-compliant examples

A compliance document that contains both a compliant and a non-compliant example in the same section causes the judge to retrieve a chunk containing both. The criterion may score 100 even when the response matches the non-compliant example. The diagnostic judge catches this, but the fix is in the document structure.

One clause per section

If a section covers multiple obligations, split it. The judge retrieves one chunk per criterion. A chunk that covers three obligations will be retrieved for all three criteria but may not contain the specific obligation each criterion is testing.

Avoid tables for policy rules

Tables in DOCX are chunked as a single block. If your document uses tables to present policy rules, each row is not independently retrievable. Convert policy tables to headed sections.

Terminology and criterion matching

The judge uses your criterion description to retrieve the relevant clause. Terminology mismatches between your criteria and your document reduce retrieval accuracy.

If your document says “personally identifiable information” and your criterion says “personal data”, retrieval may miss the right clause. The criterion is marked unscored, not scored as zero – but unscored is not passing.

Use the exact terms from the regulation. If the regulation uses “data subject”, your criterion should say “data subject.” If the document uses “PII” throughout, use “PII” in criteria. If it spells it out, spell it out.

Writing compliance criteria for this store

Four rules:

1. Write criteria as specific questions or statements, not topic labels. “GDPR Article 5” does not retrieve a useful clause. “Does the response make claims about data retention that contradict Article 5(1)(e) of the uploaded GDPR document?” does.
2. Use the same terminology as the document (see above).
3. State the requirement type explicitly – presence, accuracy, prohibition, or conditional. The judge infers type from description. Vague descriptions produce inconsistent inferences.
4. Reference the clause number if you know it. “…as defined in Section 3.2 of the uploaded policy” helps retrieval find the right chunk even when terminology is imperfect.

Attaching a compliance store to a run

Evaluations

On the evaluation form, select your compliance store from the Compliance Store dropdown (separate from the Context Store dropdown). Both can be attached to the same run. The model receives the context store (if attached). The judge uses the compliance store.

Benchmarks

Select a compliance store when creating or editing a benchmark. Saved with the benchmark config. If you re-upload a document to the store, the next run uses the updated content. Previous results are not affected.

What happens per criterion

For each criterion in your rubric, the judge:

1. Uses the criterion description to retrieve the most relevant clause from the compliance store
2. Scores the response against that clause
3. Records which clause was retrieved – visible in the Score tab

If no clause is found above the confidence threshold, the criterion is marked unscored. Unscored criteria are excluded from the aggregate score calculation. A run with many unscored criteria indicates a document structure or terminology problem, not a model problem.

Scores and what they mean

A compliance rubric score measures consistency between model outputs and your uploaded documents. It does not measure legal compliance. It does not certify anything.

• High score: the response is consistent with the clauses retrieved from your documents. The judge found no contradictions between the response and the relevant policy text.
• Low score: the response contradicts or omits content required by the retrieved clauses. Check the Score tab to see which criterion failed and which clause it was scored against.
• Unscored criterion: no clause was retrieved above the confidence threshold. This is a document or terminology problem. See the terminology section above.
• Score drift over time: if you run the same benchmark weekly and scores drop, something changed – the model, the prompt, or the document. Check Run History to identify when the drop occurred. The Diagnostics tab will flag what changed.

The diagnostic judge and compliance runs

The diagnostic judge runs automatically on every compliance evaluation. Compliance-specific findings it produces:

• Criterion text does not match document terminology – retrieval missed the relevant clause
• Mixed chunk – retrieved chunk contains both compliant and non-compliant examples
• Low confidence match – the retrieved clause is a weak match; criterion scored but reliability is low
• No clause found – criterion marked unscored; document structure or terminology problem
• Criterion description ambiguous – the judge could not infer a scoring standard from the criterion text

Check the Diagnostics tab when any criterion is unscored or when scores are higher than expected. False positives are as important as false negatives in compliance testing.

Updating your compliance store

When a regulation or policy changes:

1. Re-upload the updated document to the store (or upload a new version alongside the old one)
2. Re-run the benchmark
3. Scores reflect the new document automatically

Previous run results are not affected – they reflect the document at the time of the run.

Common mistakes

Using a context store as a compliance store

The model sees a context store. The judge uses a compliance store. If you attach your policy document as a context store, the model answers from it but the judge scores without it. Compliance scoring does not happen. Select the correct store type at creation.

Uploading a PDF when DOCX is available

The most common cause of poor clause retrieval. If the regulation exists as a Word document, use it.

One large file with all regulations

A single file containing your entire compliance library produces poor retrieval. The judge retrieves one chunk per criterion. A chunk from a mixed file may contain content from multiple regulations. Keep stores narrow – one regulation or policy area per store.

Writing criteria as topic labels

“GDPR”, “Data retention”, “Consent” – these do not retrieve useful clauses and do not give the judge a scoring standard. Write criteria as specific questions. See the criteria section above.

Expecting a score of 100 to mean compliant

The score measures consistency with your documents. It does not certify compliance. A response can score 100 and still be non-compliant if your documents are incomplete, your criteria are vague, or the wrong clause was retrieved.

Not checking the Score tab

The Score tab shows which clause was retrieved for each criterion and how the response was scored against it. This is the primary debugging tool for unexpected compliance scores. Always check it before concluding that a model is or is not compliant with a regulation.

What’s next