Jobs are async – always

Every write operation returns a job_id, not a result. There is no synchronous mode. If your code expects a result directly from POST /compare, it will only see a job_id. See Async jobs.

RAG stores must exist before upload

You cannot upload a file and associate it with a store later. The store_id must be provided at upload time (in POST /files/upload-url). If you upload without a store_id, the file is stored but never ingested – ingest_status will be none.

If you need to ingest a file into a store after the fact, delete it and re-upload with the correct store_id.

Ingestion is async – check sync_status before querying

After confirming a file upload, the store is not immediately queryable. Ingestion runs asynchronously. Using a store in a comparison or evaluation before ingestion completes will return results with no RAG context injected, with no error.

Always check sync_status: ready on the store via GET /rag/stores before using it in a run.

Use models, not providers, for explicit control

providers is a shorthand that uses your tier’s default model for each provider. If you want a specific model, use models:

{"models": {"openai": "gpt-4o-mini"}}

not:

{"providers": ["openai"]}

The default model for a provider can change when the platform registry is updated. Using models explicitly makes your integration stable.

Model IDs can change

Providers retire models and replace them with new IDs. If a model ID you are using is retired, the result for that model will have error_type: model_not_found. The job itself will still complete.

Call GET /models periodically to check for model ID changes. The response includes the current active model IDs for your tier.

Job records expire after 1 hour

The GET /jobs/{job_id} endpoint returns 404 after 1 hour. Retrieve and store results before then. Completed results are also stored in their respective tables (comparisons, evaluations, benchmark runs) with a longer retention period.

Deleting a store deletes all its files

DELETE /rag/stores/{store_id} is a cascade delete. It removes the store, all files in it, their S3 objects, and all Pinecone vectors. There is no undo. If the store is used by active benchmarks, those benchmarks will be frozen.

Benchmark runs use saved config – no overrides at run time

POST /benchmarks/{suite_id}/run uses the models and scoring config saved on the benchmark. There is no way to override models or scoring at run time. To change the config, use PATCH /benchmarks/{suite_id} first.