Background runs
Calling an agent (POST /api/client/v1/agents/responses, POST /api/client/v1/responses) normally holds the HTTP connection open until the turn is done — model round-trips, tool loops, guardrail checks, everything. That breaks down for turns that legitimately take minutes: load balancers, reverse proxies and serverless functions cap connection time, and nobody can predict before running how long a turn will take.
A background run fixes that: you say up front that the call may take a while, Console queues the turn and answers 202 Accepted with a run at once, and you poll the run or receive a callback when it finishes.
Two behaviours, deliberately
There is no clever third mode. If you do not ask for background mode, the call is synchronous and has a hard ceiling: a turn still running when it passes is terminated and the caller gets 504. Console never silently converts a slow synchronous call into a background job — the caller knows better than the server.
Choosing a mode
The caller declares background mode up front, with a header or a body field. It is a signal your code sets on its request — there is no checkbox for it in the Playground.
| Signal | Example | Applies to |
|---|---|---|
| Header | X-Cognipeer-Background: true | Every endpoint, whatever the body looks like. The canonical signal. |
| Body field | "background": true | Mirrors the OpenAI Responses background field, so an unmodified OpenAI SDK client can send it. |
| The agent's default | Build → Execution → When a call doesn't say | Used only when neither of the above said anything. |
Resolution order: header, then body, then the agent's default. The first signal that says something decides. The header counts when it is true or false (case-insensitive, whitespace ignored — anything else is ignored); the body field counts only as a JSON boolean (the string "true" is ignored). So an explicit false wins over an agent whose default is background, and X-Cognipeer-Background: false wins over "background": true in the same request.
Settings in Studio
Build → Agents → your agent → Build → Execution. These are operational settings: they apply immediately and are not part of a published version. Every number is an upper bound this agent opts into — the server applies min(environment ceiling, tenant quota, this setting), and the fields show the current ceiling as their maximum. A line at the foot reads, for example, Effective now: sync timeout 600s · background max 30 min · 10 concurrent runs per tenant.
| Section | Setting | Default | What it does |
|---|---|---|---|
| Synchronous calls | Timeout (seconds) | the ceiling, 600 s | A call that runs longer returns 504; the turn is stopped. Minimum 5. |
| Background runs | Allow background runs | on | Off: API callers sending background: true get 400 agent_background_disabled, and the agent's default mode is treated as synchronous. |
| When a call doesn't say | Run synchronously | Run synchronously or Run in background. An explicit background: true|false in the request always wins. | |
| Maximum run time (minutes) | the ceiling, 30 min | A run still going after this is stopped and marked failed (max_duration_exceeded). Minimum 1. | |
| Default callback | Callback URL | none | Used for background runs started without their own callback_url. The result is POSTed there when the run finishes, fails or is canceled. |
| Signing secret | none | 16+ characters, write-only: once saved, the page shows a mask, and sending the mask back keeps the stored value. Used only together with the default URL. |
The agent's Sessions tab has a Background runs switch next to Sessions. It lists the agent's runs (refreshing every 5 seconds while any is active) with All / Active / Failed filters: the run's status, its id with an Open conversation link, when it started, duration, the callback state (pending, delivered, failed, or polling when there is none), the error, and a Cancel run button for a queued or running run.
Environment ceilings (operator settings):
| Variable | Default | |
|---|---|---|
AGENT_SYNC_TIMEOUT_MS | 600000 (10 min) | The synchronous ceiling. Set it below the shortest infrastructure timeout in your deployment (load balancer idle timeout, API gateway timeout, function max duration) — otherwise the infrastructure answers its own 504 first and the caller sees an opaque failure with no run to check on. |
AGENT_BACKGROUND_MAX_DURATION_MS | 1800000 (30 min) | The background ceiling. |
AGENT_BACKGROUND_MAX_CONCURRENT_RUNS_PER_TENANT | 10 | Queued + running runs a tenant may hold. |
AGENT_BACKGROUND_MAX_CONCURRENT_RUNS_PER_PROJECT | 0 (no cap) | The same per project. |
AGENT_RUN_CONCURRENCY | 4 | Runs a node works on at once. |
AGENT_RUN_RETENTION_DAYS | 30 | Runs (and their message and result) are kept this long. |
AGENT_RUN_HEARTBEAT_INTERVAL_MS, AGENT_RUN_HEARTBEAT_STALE_MS | 15000, 45000 | How often a worker proves it is alive, and when a silent run is declared lost. |
AGENT_RUN_REQUEUE_AFTER_MS | 120000 | A queued run no worker picked up is queued again after this. |
Start a run
Accepted by POST /api/client/v1/responses (draft) and POST /api/client/v1/agents/responses (published), alongside the usual model / input / previous_response_id / session_id / version / runtime_context fields:
| Where | Name | Notes |
|---|---|---|
| Header | X-Cognipeer-Background | true | false. Highest precedence. |
| Body | background | boolean. |
| Body | callback_url | Background only (ignored on synchronous calls). http/https, at most 2048 characters, must resolve to a public address — otherwise 400 invalid_callback. When omitted, the agent's own default callback is used. |
| Body | callback_secret | Background only; needs callback_url (400 otherwise). 16–256 characters. Signs each delivery. A callback_url sent without a secret is delivered unsigned — the agent's default secret goes only with the agent's default URL. |
| Header | Idempotency-Key | Background only, at most 255 characters. |
curl -X POST "$CONSOLE_URL/api/client/v1/agents/responses" \
-H "Authorization: Bearer $COGNIPEER_API_TOKEN" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: 3f1c9a2e-7b1d-4c55-9a51-0d6c2f1e8b7a" \
-d '{
"model": "support-bot",
"input": "Research this topic in depth and summarize the findings",
"background": true,
"callback_url": "https://your-server.com/webhooks/agent-run",
"callback_secret": "a-shared-secret-of-16-plus-chars"
}'const run = await client.agents.responses.create(
{
model: 'support-bot',
input: 'Research this topic in depth and summarize the findings',
background: true,
callback_url: 'https://your-server.com/webhooks/agent-run',
callback_secret: 'a-shared-secret-of-16-plus-chars',
},
{ idempotencyKey: '3f1c9a2e-7b1d-4c55-9a51-0d6c2f1e8b7a' },
);
const done = await client.agents.runs.wait(run.id, { timeoutMs: 10 * 60_000 });
if (done.status === 'succeeded') console.log(done.result?.output[0]?.content[0]?.text);Either call answers 202 Accepted with the run (200 instead, with the existing run, for a replayed Idempotency-Key):
{
"id": "run_65f1c0a2b9d3e4f5a6b7c8d9",
"object": "agent.run",
"status": "queued",
"stop_reason": null,
"agent": "support-bot",
"conversation_id": "6f1c0a2b9d3e4f5a6b7c8d9e",
"result": null,
"error": null,
"created_at": 1760000000,
"started_at": null,
"completed_at": null,
"cancel_requested_at": null,
"max_duration_ms": 1800000,
"callback": { "url": "https://your-server.com/webhooks/agent-run", "status": "pending", "attempts": 0, "signed": true }
}Forcing a synchronous call on an agent whose default is background:
curl -X POST "$CONSOLE_URL/api/client/v1/responses" \
-H "Authorization: Bearer $COGNIPEER_API_TOKEN" -H "Content-Type: application/json" \
-d '{ "model": "support-bot", "input": "Quick question", "background": false }'Poll, wait, cancel
GET /api/client/v1/agents/runs/{runId} # status + result
POST /api/client/v1/agents/runs/{runId}/cancel # cooperative cancel (queued or running)Both resolve the run through the caller's tenant and project — a token cannot read or cancel a run of another project, even if it guesses the id. {runId} is run_{id} as returned; the resp_{id} form of a finished run's result is accepted too.
| Field | |
|---|---|
status | queued, running, requires_action, succeeded, failed, canceled. |
stop_reason | How the agent turn ended — end_turn, limit, requires_action, cancelled, error, budget_reached. null while active. A run that a limit stopped still ends succeeded; stop_reason says how. |
result | The Responses body a synchronous call would have returned — once succeeded, and while requires_action (with required_action). |
error | { type, message } once failed or canceled. |
created_at, started_at, completed_at, cancel_requested_at | Epoch seconds. |
callback | null when there is none; otherwise url, status (pending, delivered, failed), attempts and signed. |
error.type is one of agent_error, worker_lost, max_duration_exceeded, canceled_by_caller, precondition_failed (the agent or token is no longer valid when the run starts), owner_disabled and owner_unverifiable (the user who queued the run has since been disabled, or could not be verified — a run never executes on a disabled user's authority).
When the agent has structured output, result carries two additive fields: output_parsed (the validated object — only when parsing and schema validation succeeded) and output_error (why it failed). A structured-output failure does not change the status: the run is still succeeded, so check result.output_error.
# Poll until a terminal state — and stop on requires_action too: it waits for a person.
while true; do
RUN=$(curl -s "$CONSOLE_URL/api/client/v1/agents/runs/$RUN_ID" -H "Authorization: Bearer $COGNIPEER_API_TOKEN")
STATUS=$(echo "$RUN" | jq -r .status)
case "$STATUS" in succeeded|failed|canceled|requires_action) break ;; esac
sleep 2
done
echo "$RUN" | jq .
# Cancel
curl -X POST "$CONSOLE_URL/api/client/v1/agents/runs/$RUN_ID/cancel" -H "Authorization: Bearer $COGNIPEER_API_TOKEN"const run = await client.agents.runs.get(runId);
// wait() returns on succeeded, failed, canceled — and on requires_action, which waits on a person.
const settled = await client.agents.runs.wait(runId, { pollIntervalMs: 2000, timeoutMs: 15 * 60_000 });
await client.agents.runs.cancel(runId);Cancellation is cooperative. The cancel call answers 200 with the run, which may still read running for about a second until the worker observes the request; the run is finalized canceled only once the turn has actually stopped, and what the cancelled turn would have written to the conversation is discarded. Until then the run keeps the conversation reserved, so a message sent in between gets the usual 409. A cancel that arrives after the run has already finished is 409 agent_run_already_terminal.
What "cancelled" and "timed out" do not promise
Cancellation and the deadline stop the agent loop from starting its next step. They do not interrupt a tool call or a model call already in flight: a tool that is mid-way through sending an email will finish. "Terminated" means no further steps run and the caller stops waiting — not that a side effect in progress is guaranteed to be interrupted.
One active run per conversation
A conversation can have at most one queued or running run, and the same rule applies to synchronous calls: a request against a conversation that already has an active run — background or synchronous — is refused with 409 agent_run_conflict before any turn starts. Two turns writing the same conversation at once would silently erase one of them. The check is atomic (an insert guarded by a unique index), not a read followed by a write. Deleting a session with an active run is refused the same way.
Other background-specific refusals: 400 agent_background_disabled (the agent turned background runs off), 429 agent_run_concurrency_limit (error.type: rate_limit_error; the tenant or project already has the configured number of runs queued or running), and a 429 spend_limit_exceeded up front when the tenant's spend limit is used up — you get it at once, not after a 202.
Idempotency
Idempotency-Key is a background-mode concept. A synchronous request carrying it is rejected — 400 idempotency_key_sync_not_supported — because a synchronous call persists nothing to key a retry against; honoring it there would look like it works and do nothing.
On a background call, the key is looked up within your tenant and project. The same request (agent, conversation, message, version and the file ids you named) returns the existing run — 200, whatever its status. A different request under the same key is 409 idempotency_key_conflict. This prevents a retried request from repeating tool side effects; it does not undo side effects of an earlier attempt made without a key.
The synchronous ceiling
A synchronous call that passes its ceiling is answered 504 immediately:
{
"error": {
"type": "timeout",
"code": "timeout",
"message": "The agent turn exceeded the synchronous timeout and was terminated. Side effects from tool calls already in flight may have occurred; this request was not retried automatically.",
"side_effects_possible": true,
"retryable": false
}
}It is 504 (not 408, which means a slow client and is often auto-retried). The turn is not converted into a run, gets no run id and is not retried; if the model call it was waiting on later finishes, its result is discarded — it is never written to the conversation. Do not blindly retry: the turn may already have sent an email or written a record.
Callbacks
When a run has a callback, Console POSTs to it when the run finishes, fails, is canceled or pauses for a person:
POST /webhooks/agent-run
content-type: application/json
user-agent: cognipeer-agent-runs/1.0
x-cognipeer-event: agent_run.succeeded
x-cognipeer-event-id: evt_65f1c0a2b9d3e4f5a6b7c8d9_succeeded
x-cognipeer-signature: t=1760000042,v1=…{
"id": "evt_65f1c0a2b9d3e4f5a6b7c8d9_succeeded",
"event": "agent_run.succeeded",
"createdAt": "2026-10-11T09:15:02.000Z",
"runId": "run_65f1c0a2b9d3e4f5a6b7c8d9",
"conversationId": "6f1c0a2b9d3e4f5a6b7c8d9e",
"data": { "result": { "id": "resp_65f1c0a2b9d3e4f5a6b7c8d9", "object": "response", "status": "completed", "…": "…" } }
}event | data |
|---|---|
agent_run.succeeded | { result } — the Responses body |
agent_run.failed | { errorReason, message? } |
agent_run.canceled | {} |
agent_run.requires_action | { result } — carrying required_action |
The event id is stable per run and event, so a receiver can deduplicate retries and redeliveries on it. The signature is HMAC-SHA256(secret, "<t>.<raw body>"), the same scheme as webhooks — and the verifyAgentRunCallback helper in @cognipeer/console-sdk/webhooks (an alias of verifyWebhook). Delivery has a 10 second timeout and is retried by the queue up to 5 times with exponential backoff (2, 4, 8, 16, 32 seconds); the run's callback.status and attempts are stored on the run, so a restart mid-retry does not lose the notification. A callback that never settles is picked up again by the reconciler. Callbacks fire for failed and canceled outcomes too, not only for success.
If you would rather not run an endpoint per call, subscribe a webhook to agent_run.* instead: it hears about every run transition, whether or not the run carries a callback.
If a worker dies
A periodic reconciler (every 30 seconds) looks for running runs whose heartbeat has gone stale. Such a run is finalized failed with error.type: worker_lost and its failure callback fires. It is not restarted automatically: an agent turn may already have called a tool with real side effects, and re-running it would repeat them. A queued run that never started is safe to queue again and is. If the process dies mid synchronous call, the reservation it held is cleared, so the conversation is not wedged.
Security notes
- Runs are project-scoped.
GETandcancelresolve the run through tenant and project; a token from another project in the same tenant gets404even with the right id. - A run never executes on a disabled user's authority. If the user who queued it was disabled in the meantime (or cannot be verified), the run fails with
owner_disabled/owner_unverifiableinstead of running. - Callback URLs are validated twice. Submitting one that does not resolve to a public address is
400 invalid_callback; delivery then re-resolves, pins the address and re-checks every redirect hop, so a name that rebinds to an internal service later is still refused. - Callbacks should be signed. A
callback_urlsent without a secret is delivered unsigned. Sendcallback_secret(16–256 characters), or set the agent's default callback secret, and verify the signature. - Idempotency keys are scoped to your tenant and project, and compared against the whole request — not just the key.
Troubleshooting
409 agent_run_conflict when I send a message. A run is queued or running on the conversation. Wait for it, or cancel it. A run that is requires_action does not hold the conversation in the same way — answer its requests instead (approvals).
The run sits in queued. No worker has picked it up: check that the node serving the queue is running. After AGENT_RUN_REQUEUE_AFTER_MS the reconciler queues it again.
504 on a call I expected to finish. It passed AGENT_SYNC_TIMEOUT_MS (or the agent's lower Timeout setting). Use background mode for turns that can take minutes.
My callback never arrives. Check callback.status on the run: failed means all five attempts were refused. The URL must be public and answer 2xx within 10 seconds.
I polled until "terminal" and my loop never ended. requires_action is non-terminal but waits for a person. Stop polling on it, too.
What's new
Run status now includes requires_action; callbacks fire for it (agent_run.requires_action); a run can be continued after an approval or a raised session budget; every run transition also reaches webhook subscriptions; and the run resource reports stop_reason, max_duration_ms and the callback state.