Replays
Updated on 20 November 2026.
Replays answer "what would happen if" with your own traffic. Pick requests the LLM Gateway has already logged, send them again, and compare the new answers with the originals before you change anything for real:
- Would a cheaper model answer as well?
- Does the model we use today still answer the way it did?
- What would this routing rule save, and what would it cost in quality?
- Does token saving change the answers?
A judge model you pick reads each pair of answers and says whether the new one is better, equivalent or worse. Results are re-weighted from the sample to your real mix of requests, so a share you read is a share of your traffic.
Where to find it
Open Settings > AI Gateway > LLM Gateway and select the Model Ops tab. Replays lists every run, newest first, with its progress while it runs.
You can also start from Logs:
- Replay these in the Logs header replays the requests that match your current filter.
- Replay… in a request's detail drawer replays that one request.
Kinds of replay
Every replay is a run, including a single request. Pick the kind when you start it:
Same model, now is the control. Answers vary from one call to the next even on the same model, so the control shows how much of a difference is just that. When the control itself scores low, the model may have changed since the request was first logged.
To set one up, see Set up a replay. To read the results, see Read replay results.
How a replay runs
- Requests are sent exactly as recorded. Only the model changes, or token saving in a token saving test. You can override the temperature and the maximum output tokens; nothing else in the request can be edited.
- The source application's content policies apply. Redaction happens before the request leaves, and each answer is checked on the way back. Access, quota, budget, rate-limit and routing rules are skipped, and there is no failover or response cache.
- Any enabled model on your providers can be a target or the judge. It does not have to be attached to an application.
- Your providers bill the calls. Replays never draw on an application's quota or budget. Each run has its own spend cap and stops when it reaches it.
- Runs keep going in the background. You can leave the page; the list and the run page show progress while it runs. Requests are replayed in a shuffled order, so the results so far are a fair sample at any point.
- When a target model rejects a setting it does not support, the request is sent again without it, and the answer shows what was adjusted and why. For example, a model without adaptive thinking gets a fixed thinking budget.
Limits
The Replays page states the limits for your organization. Today they are:
Some requests cannot be replayed, and the set-up page counts them before you start:
Replays in Logs and analytics
Replay and judge calls are real gateway requests. They belong to the source application, with the admin who started the run as the user, so they appear in Logs and are counted in analytics and spend.
- In Logs, they carry a Replay or Replay judge tag under the status. Filter on them with Replay calls under the more filters.
- A replay's detail drawer links back to its run and to the request it replayed. An original that was replayed lists the runs that replayed it.
- Deleting a run deletes its results. The replayed calls stay in Logs as ordinary requests.
Who can use it
Starting, stopping and deleting a run is recorded in the audit log with its source, picking, models, judge and spend cap.