Async Jobs

Why async

Every write operation in LLM Prover – comparisons, evaluations, benchmark runs – calls one or more LLM providers in parallel. Provider calls take 1-30 seconds depending on model, prompt length, and provider load. Holding an HTTP connection open for that duration is unreliable and wastes resources on both sides.

Instead, every write endpoint returns a job_id immediately (HTTP 202) and processes the work asynchronously. You poll a separate endpoint to check status and retrieve the result when it is ready.

The pattern

Every async operation follows the same three steps:

1. Submit – POST to the endpoint. Receive a job_id.

curl -X POST https://api.llmprover.pysolvr.com/compare \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "...", "models": {"openai": "gpt-4o"}}'

Response:

{"ok": true, "data": {"job_id": "a1b2c3d4-...", "status": "pending"}}

2. Poll – GET /jobs/{job_id} until status is complete or failed.

curl https://api.llmprover.pysolvr.com/jobs/a1b2c3d4-... \
  -H "Authorization: YOUR_API_KEY"

3. Retrieve – The full result is in data.result when status is complete. No second request needed.

Job status values

Polling rules

Poll no faster than once every 2 seconds. Polling faster than this wastes your rate limit budget and does not make jobs complete sooner.

Typical completion times:

Stop polling after 5 minutes. If a job has not completed by then, treat it as failed and surface an error to the user.

Python polling loop

import time
import requests

def poll_job(job_id, api_key, base_url="https://api.llmprover.pysolvr.com", timeout=300):
    headers = {"Authorization": api_key}
    deadline = time.time() + timeout
    while time.time() < deadline:
        time.sleep(2)
        resp = requests.get(f"{base_url}/jobs/{job_id}", headers=headers).json()
        status = resp["data"]["status"]
        if status == "complete":
            return resp["data"]["result"]
        if status == "failed":
            raise RuntimeError(f"Job failed: {resp['data'].get('error')}")
    raise TimeoutError(f"Job {job_id} did not complete within {timeout}s")

Job record expiry

Job records expire after 1 hour. Retrieve and store results before then. Once a job record expires, the result is gone – it cannot be recovered from the jobs endpoint.

Completed results are also stored in their respective tables (comparisons, evaluations, benchmark runs) with a longer retention period based on your tier. Use GET /compare/{comparison_id} or GET /benchmarks/{suite_id}/runs to retrieve stored results after the job record has expired.

What’s next

• Rate limits – how polling counts against your limits
• Errors – handling failed jobs