Incidents API

Fast controls for a provider outage. Mark a provider (or one of its models) down for every app or chosen apps, and the gateway stops routing and failing over to it until it is restored or the mark-down expires. Incident controls are not policy changes: they work whether or not the Policy Engine is on, publish no revision, and never touch your routes.

Action	Effect	Scope
Preview	Which apps and routes use the provider, routes that would be left with no target, and where the rest would fail over. Writes nothing.	`incidents:write`
Mark down	Takes the provider out of routing and failover for the chosen apps until an expiry, at most 72 hours ahead. Conversations pinned to it route again on their next request.	`incidents:write`
Status	Whether the latest change is in force on the gateway you called, and by when every gateway applies it.	`incidents:read`
Restore	Removes the mark-down.	`incidents:write`
Reset cooldowns	Gateways forget the cooldown and circuit-breaker state of the chosen targets, so a recovered provider takes traffic at once.	`incidents:write`
Drop pins	Ongoing conversations of the chosen apps or targets route again on their next request.	`incidents:write`

Reads accept incidents:read, read or incidents:write. Try the runbook in the Management API playground.

Outage runbook

1. Preview

mark-downs:preview

Routes left without targets

2. Mark down

apps "*" or names

expires_in_seconds ≤ 72 h

3. Confirm

incidents/status

Applied everywhere by

4. Restore

DELETE the mark-down

Optional: reset cooldowns

QuilrAI

export API="$MANAGEMENT_URL/llmgateway/management/v1"
auth=(-H "Authorization: Bearer $MANAGEMENT_KEY" -H 'Content-Type: application/json')
BODY='{"provider_label": "azure-eu", "apps": "*", "reason": "Azure EU outage", "ticket": "INC-123", "expires_in_seconds": 14400}'

# 1. Preview: what loses its target, and where the rest fails over
curl -X POST "$API/incidents/mark-downs:preview" "${auth[@]}" -d "$BODY" \
  | jq '{apps_affected, routes_losing_all_candidates, fallback_destinations}'

# 2. Mark down for every app, for 4 hours. The id is yours; reuse it to change or restore.
curl -X PUT "$API/incidents/mark-downs/inc-123-azure-eu" "${auth[@]}" \
  -H "Idempotency-Key: inc-123-down" -d "$BODY" | jq '{changed, mark_down, propagation}'

# 3. Confirm it is in force
curl "$API/incidents/status" "${auth[@]}" | jq '{record_version, this_server, fleet}'

# 4. When the provider recovers: restore, and let gateways retry it at once
curl -X DELETE "$API/incidents/mark-downs/inc-123-azure-eu" "${auth[@]}"
curl -X POST "$API/incidents/reset-cooldowns" "${auth[@]}" -H "Idempotency-Key: inc-123-reset" \
  -d '{"apps": "*", "provider_label": "azure-eu"}'

Restoring does not move conversations back: conversations that moved during the incident stay on their new target until they go idle. To rebalance them, drop pins for the provider's apps.

Mark-downs

Body. provider_label (a shared provider or an app's provider label, else 422 unknown_provider_label), optional model (null marks every model), apps ("*" for every app now and later, or app names, else 422 unknown_app), reason, optional ticket, and an expiry.
Expiry is required. Send expires_at (RFC 3339 with a timezone, or epoch seconds) or expires_in_seconds, not both. It must be in the future and at most 72 hours ahead (422 expiry_too_far); reads show expires_in (seconds left). An expired mark-down is no longer listed and no longer applies. Extend a long incident with another PUT before it expires.
Ids are yours. PUT /incidents/mark-downs/{id} with ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$ creates (201) or replaces (200). The same body again is 200 changed: false. Changing an existing one needs If-Match with its version (428 / 412).
Retries are safe. With an Idempotency-Key, retrying the PUT that set the current mark-down returns the same result (replayed: true) even with expires_in_seconds, which would otherwise compute a new expiry. Without a key, a new expiry is a change and needs If-Match.
Restore. DELETE /incidents/mark-downs/{id}; If-Match is optional and checked when sent. An id that is absent (never set, already restored or expired) is 404 mark_down_not_found, which a restore script can treat as success.
Limits. At most 64 active mark-downs per tenant (409 too_many_mark_downs).
Console rows. Per-app mark-downs set in the console are listed read only, with source: "app".

Reset cooldowns and drop pins

Both take {"apps": "*" | [names], "provider_label"?, "model"?} and require an Idempotency-Key; a retry returns the original result without acting again.

Without provider_label, every target of the selected apps is reset (or every pin of those apps dropped).
With provider_label (and optionally model, which needs provider_label), only those targets are affected. applies_to in the response says what was covered.
Drop pins covers every kind of conversation pin, including pins set by routing groups, model lists and earlier failovers.

Propagation

Writes go to the central configuration and then to the gateway you called. Every other gateway applies them on its next configuration sync, within about 65 seconds. GET /incidents/status?record_version=N (default: the current version) reports:

Field	Meaning
`this_server.applied`	The gateway you called runs this version.
`fleet.bound_seconds`	How long after a write every gateway is guaranteed to apply it (about 65).
`fleet.applied_everywhere_by`	The time by which every gateway applies it. Other gateways are not polled, so `fleet.confirmed` is always `false`.

Write responses carry propagation with the record_version to check. The preview reports health and cooldowns from the gateway you called (health_scope: "this_node").

Errors

Status	Codes
400	`invalid_request` (body, id, expiry or `record_version` shape; `model` without `provider_label`), `idempotency_key_required`
403	`insufficient_scope`
404	`not_found` (read of an absent mark-down), `mark_down_not_found` (restore of an absent one), `record_version_not_found`
409	`idempotency_conflict`, `idempotency_replay_expired`, `too_many_mark_downs`
412	`version_conflict`
422	`expiry_too_far`, `unknown_provider_label`, `unknown_app`
428	`precondition_required`
503	`central_service_required` (a self-hosted node connected to a QuilrAI-hosted service; call the regional host)

Endpoints

POST/incidents/drop-pins

Drop session pins

Every session pin (any kind) of the selected apps becomes invalid, so the next request of each conversation routes again. Same body, scoping and Idempotency-Key replay as reset-cooldowns; `provider_label` (+ `model`) narrows it to pins on those targets.

REQUIRESincidents:write

Path, query & header parameters 1

Idempotency-Key (header)stringrequired

Unique per logical write. A retry with the same key and body replays the original result for 24 hours; the same key with a different body is a 409.

min length: 1max length: 200

Full request body specification application/json

appsone of 2 variantsrequired

Every app now and later ("*"), or these apps (names, case-insensitive).

Variant 1 "*"

"*"

Variant 2 array<string>

array<string>

min items: 1max items: 64

modelstring | nulloptional

With provider_label: only this model of it.

max length: 200

provider_labelstring | nulloptional

Narrows the action to targets with this provider label in the selected apps; omitted: every target of the selected apps.

max length: 200

Unknown fields are rejected in this object.

curl --request POST \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/drop-pins' \
  --header 'Authorization: Bearer <management-key>' \
  --header 'Idempotency-Key: create-support-bot-001' \
  --header 'Content-Type: application/json' \
  --data '{
  "apps": [
    "Support Bot"
  ]
}'

Authentication, errors & retry rules Try it in the playground

GET/incidents/mark-downs

List mark-downs

Active tenant mark-downs (expired ones are not listed) and the tenant-level cooldown and pin generations. Per-app mark-downs set in the console are listed too, read-only, with source "app".

REQUIRESincidents:read

No request body.

curl --request GET \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/mark-downs' \
  --header 'Authorization: Bearer <management-key>'

Authentication, errors & retry rules Try it in the playground

DELETE/incidents/mark-downs/{mark_down_id}

Restore a provider

Removes the mark-down. If-Match is optional and checked when sent. An absent id is 404 mark_down_not_found, which scripts may treat as success (already restored or expired).

REQUIRESincidents:write

Path, query & header parameters 2

mark_down_id (path)stringrequired

Client-chosen mark-down ID, ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$.

If-Match (header)stringoptional

Quoted mark-down version. Required on PUT only to change an existing mark-down (the same body again is a no-op); optional on DELETE, checked when sent.

Full request body specification application/json

Unknown fields are rejected in this object.

curl --request DELETE \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/mark-downs/inc-123-azure-eu' \
  --header 'Authorization: Bearer <management-key>' \
  --header 'If-Match: "3f1c0a9e5b7d"'

Authentication, errors & retry rules Try it in the playground

GET/incidents/mark-downs/{mark_down_id}

Read a mark-down

One active mark-down with its propagation state. `version` is its ETag.

REQUIRESincidents:read

Path, query & header parameters 1

mark_down_id (path)stringrequired

Client-chosen mark-down ID, ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$.

No request body.

curl --request GET \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/mark-downs/inc-123-azure-eu' \
  --header 'Authorization: Bearer <management-key>'

Authentication, errors & retry rules Try it in the playground

PUT/incidents/mark-downs/{mark_down_id}

Mark a provider down

Creates (201) or replaces (200) the mark-down with this client-chosen ID. Sending the same body again is a no-op (200, changed false); changing an existing mark-down requires If-Match with its version. `provider_label` must be a tenant provider or an app's provider label; `apps` is "*" (every app, including apps created later) or app names. `expires_at` is required, at most 72 hours ahead, and enforced on read; `expires_in_seconds` may be sent instead. A retry with the Idempotency-Key of the PUT that set the mark-down replays its result. A mark-down also invalidates session pins on that target. Applies whether or not policy governs the app, at most 64 active per tenant.

REQUIRESincidents:write

Path, query & header parameters 2

mark_down_id (path)stringrequired

Client-chosen mark-down ID, ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$.

If-Match (header)stringoptional

Quoted mark-down version. Required on PUT only to change an existing mark-down (the same body again is a no-op); optional on DELETE, checked when sent.

Full request body specification application/json

appsone of 2 variantsrequired

Every app now and later ("*"), or these apps (names, case-insensitive).

Variant 1 "*"

"*"

Variant 2 array<string>

array<string>

min items: 1max items: 64

expires_atone of 2 variantsoptional

RFC 3339 time with a timezone, or epoch seconds; in the future and at most 72 hours ahead. Required unless expires_in_seconds is sent.

Variant 1 string

string

format: date-time

Variant 2 number

number

expires_in_secondsnumberoptional

Alternative to expires_at (not both): seconds from now, at most 72 hours.

max: 259200greater than: 0

modelstring | nulloptional

Only this model of the provider; null marks every model.

max length: 200

provider_labelstringrequired

A tenant provider label or a provider label of an app.

min length: 1max length: 200

reasonstringrequired

min length: 1max length: 500

ticketstring | nulloptional

max length: 200

Unknown fields are rejected in this object.

curl --request PUT \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/mark-downs/inc-123-azure-eu' \
  --header 'Authorization: Bearer <management-key>' \
  --header 'If-Match: "3f1c0a9e5b7d"' \
  --header 'Content-Type: application/json' \
  --data '{
  "apps": "*",
  "expires_at": "2026-10-28T18:00:00Z",
  "model": null,
  "provider_label": "azure-eu",
  "reason": "Azure EU outage",
  "ticket": "INC-123"
}'

Authentication, errors & retry rules Try it in the playground

POST/incidents/mark-downs:preview

Preview a mark-down

Takes the same body as the mark-down PUT (expires_at optional) and writes nothing. Returns the apps whose routes use the provider, the routes left with no usable fallback, and where the others would fail over. Health and cooldowns are this server's view; traffic share is not computed.

REQUIRESincidents:write

Full request body specification application/json

appsone of 2 variantsrequired

Every app now and later ("*"), or these apps (names, case-insensitive).

Variant 1 "*"

"*"

Variant 2 array<string>

array<string>

min items: 1max items: 64

expires_atone of 2 variantsoptional

RFC 3339 time with a timezone, or epoch seconds; in the future and at most 72 hours ahead. Required unless expires_in_seconds is sent.

Variant 1 string

string

format: date-time

Variant 2 number

number

expires_in_secondsnumberoptional

Alternative to expires_at (not both): seconds from now, at most 72 hours.

max: 259200greater than: 0

modelstring | nulloptional

Only this model of the provider; null marks every model.

max length: 200

provider_labelstringrequired

A tenant provider label or a provider label of an app.

min length: 1max length: 200

reasonstringrequired

min length: 1max length: 500

ticketstring | nulloptional

max length: 200

Unknown fields are rejected in this object.

curl --request POST \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/mark-downs:preview' \
  --header 'Authorization: Bearer <management-key>' \
  --header 'Content-Type: application/json' \
  --data '{
  "apps": "*",
  "expires_at": "2026-10-28T18:00:00Z",
  "model": null,
  "provider_label": "azure-eu",
  "reason": "Azure EU outage",
  "ticket": "INC-123"
}'

Authentication, errors & retry rules Try it in the playground

POST/incidents/reset-cooldowns

Reset cooldowns

Every gateway server forgets the cooldown and circuit-breaker state of the selected apps' targets on its next sync. `provider_label` (+ `model`) narrows the reset to those targets; without it, every target of the selected apps is reset. A retry with the same Idempotency-Key returns the original result without resetting again.

REQUIRESincidents:write

Path, query & header parameters 1

Idempotency-Key (header)stringrequired

Unique per logical write. A retry with the same key and body replays the original result for 24 hours; the same key with a different body is a 409.

min length: 1max length: 200

Full request body specification application/json

appsone of 2 variantsrequired

Every app now and later ("*"), or these apps (names, case-insensitive).

Variant 1 "*"

"*"

Variant 2 array<string>

array<string>

min items: 1max items: 64

modelstring | nulloptional

With provider_label: only this model of it.

max length: 200

provider_labelstring | nulloptional

Narrows the action to targets with this provider label in the selected apps; omitted: every target of the selected apps.

max length: 200

Unknown fields are rejected in this object.

curl --request POST \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/reset-cooldowns' \
  --header 'Authorization: Bearer <management-key>' \
  --header 'Idempotency-Key: create-support-bot-001' \
  --header 'Content-Type: application/json' \
  --data '{
  "apps": "*",
  "provider_label": "azure-eu"
}'

Authentication, errors & retry rules Try it in the playground

GET/incidents/status

Incident propagation status

Whether incident record version `record_version` (default: the current one) is in force on this server: its stored copy and this process. Other nodes are not polled; each applies a version within `fleet.bound_seconds` (about 65 s) of its write, stated as `fleet.applied_everywhere_by`.

REQUIRESincidents:read

No request body.

curl --request GET \
  'https://guardrails-usa-1.quilr.ai/llmgateway/management/v1/incidents/status' \
  --header 'Authorization: Bearer <management-key>'

Authentication, errors & retry rules Try it in the playground

Routing and failover - cooldowns, failover tiers and conversation pinning.
Routing and failover API - change routes permanently, with revision history.
Management API - authentication, scopes and retry rules.

Outage runbook​

Mark-downs​

Reset cooldowns and drop pins​

Propagation​

Errors​

Endpoints​

Related​

Outage runbook

Mark-downs

Reset cooldowns and drop pins

Propagation

Errors

Endpoints

Related