AI outputs need to answer to your documents.
Clause-level compliance scoring against your own documents. Up and running in minutes.
Warning: LLM Prover measures consistency with the documents you upload. It does not provide regulatory certification, legal advice, or a compliance audit. Scores are evidence to support your compliance process, not a substitute for it.
Which model is safest for this regulated task?
Pro+Same rubric, same prompt, four models. GPT-4o Mini scored 50: Availability and Superlatives both failed. GPT-4o hit 100 at 32x the cost. The benchmark gives you the evidence to make that tradeoff deliberately.
Compliance score · 4-criterion GDPR rubric
Every score traced to a specific clause.
Pro+The result shows the score, the judge’s reasoning, and the clause retrieved from your document. If no matching clause is found, the criterion is marked unscored, not zero. An unscored criterion means your document or criteria need attention. A zero means the model failed. No silent passes.
Compliance score per criterion (GPT-4o Mini, weighted total: 75)
A/B test your regulatory documents.
Pro+When your legal team updates the policy document, does the new version produce better compliance scores? Same model, same prompt, swap the store.
Compliance score · policy document A/B (same model, same prompt)
Silent weight updates break compliance posture.
Pro+Providers update model weights without announcing it. On run 5, GPT-4o compliance dropped from 91 to 61. Art. 22 fell 57 points. With a threshold set, the alert fired before any affected output reached production. See the Compliance Store Guide.
Compliance score across benchmark runs (illustrative)
Diagnostic findings drive compliance alignment against your standards.
Pro+When a score falls short, the diagnostic judge identifies which pipeline stage caused it and gives a specific fix. Tighten context, parameters, and criteria until the score reflects your actual standard.
Diagnostic finding · context
high
stage: context
Observation: Retrieved chunk contains both a consent definition and an invalid consent example. The judge scored against the invalid example.
gdpr-articles-5-7.docx · Art. 7(1) (relevance: 0.87)
Fix: Separate the consent definition and the invalid consent example into distinct headed sections.
Diagnostic finding · parameters
medium
stage: parameters
Observation: Temperature 0.7 introduces variance in clause interpretation across runs.
temperature=0.7, seed=none
Fix: Set temperature=0.0 and a fixed seed for deterministic compliance scoring.
Make your compliance scores auditable.
Pro+At default temperature, the same compliance prompt produces different scores on every run. Set temperature to 0.0 and a fixed seed in the inference parameters when creating your benchmark. Results become deterministic: the same prompt against the same document returns the same score every time. That is a score you can show an auditor.
Score stability: locked parameters vs defaults (5 runs, same prompt)
An audit trail, not just a score.
Pro+Every run is timestamped. Every score links to the clause it was scored against. Every finding is logged. Export any run as CSV or JSON: each row contains the criterion, the retrieved clause, the score, and the judge’s reasoning. Structured evidence your compliance team can work with.
Compliance score · run history (gdpr-check-v1)
Score against your documents and answer from them.
Pro+If you want the model to answer from your policy documents and score against them in the same run, attach two stores: a context store so the model reads the documents, and a compliance store so the judge scores against them. Same documents, different roles. The model never sees the compliance store.
New evaluation
privacy-policy-v3 model reads this
gdpr-articles-5-7 judge scores against this
Evaluation result
| Model | Quality | Cost |
|---|---|---|
| GPT-4o | 88 | $0.00421 |
| Claude Sonnet 4.5 | 91 | $0.00614 |
Claude Sonnet 4.5 · compliance scores
| Criterion | Score | Retrieved clause |
|---|---|---|
| Retention limits | 94 | Art. 5(1)(e): kept no longer than necessary |
| Data subject rights | 88 | Art. 17(1): right to erasure on request |
Questions
Does LLM Prover certify that my outputs are compliant?
No. LLM Prover scores model outputs against the documents you upload and surfaces evidence of consistency or inconsistency with specific clauses. It does not provide regulatory certification, legal advice, or a compliance audit. You remain responsible for regulatory interpretation and compliance sign-off.
Which regulations does it support?
Any regulation or policy document you can upload: GDPR, HIPAA, the FCA handbook, internal policies, operational standards. LLM Prover does not have built-in knowledge of specific regulations. It scores against the document you provide, which means scores always reflect your version of the document, not a generic interpretation. If it can be uploaded, it can be scored against.
What happens if the judge cannot find a matching clause?
The criterion is marked unscored, not scored as zero. An unscored criterion is surfaced explicitly in the result. To resolve it: check that your store contains the relevant clause, tighten the criterion wording to be more specific, or restructure the document so the clause is clearly delimited. There are no silent passes.
What document formats work best?
DOCX and Markdown produce the most accurate retrieval because they are chunked by heading structure, which preserves clause boundaries. PDF, plain text (.txt), and other formats use best-effort chunking and section boundaries may not be preserved. For compliance documents where accurate clause retrieval is critical, upload as DOCX where possible.
Where to start
Compliance rubric scoring is a Pro+ feature. Pro can run scheduled benchmarks but cannot attach a compliance store. Pro+ gives you up to 12 criteria per rubric, 5 stores shared across context and compliance, and files up to 10MB. Enterprise removes all limits.
Pro $129/mo | Pro+ $199/mo | Enterprise $499/mo | |
|---|---|---|---|
| Compliance rubric scoring | -- | ✓ | ✓ |
| Criteria per rubric | -- | 12 | 20 |
| Stores (shared pool) | -- | 5 total | unlimited |
| File size limit | -- | 10MB | 25MB |
| Benchmark schedule | daily | 4x daily | hourly |
| Export results | -- | CSV, JSON | CSV, JSON |
| Export includes | -- | criterion, clause, score, reasoning | criterion, clause, score, reasoning |
Ready to measure your compliance posture?
Pro+ and above. Upload your documents and run your first compliance evaluation in minutes.