Token saving
Updated on 3 November 2026.
Reduce tokens sent to the provider with request compression and tool-output cleanup. The Policy Engine can also reuse an eligible cached response. Your application keeps using its existing API.
For optimization from your own application code, use standalone SDK token savings. It accepts a text string and an explicit choice of seven methods, returns the final text and savings estimates, and supports deterministic, cache-friendly prompt preparation without routing an LLM call through the gateway.
How the LLM Gateway and MCP Gateway rewrite JSON, HTML, Markdown and verbose text into fewer tokens before it reaches the model.
YouTube All videosConfigure this control
Choose the mode used by your gateway. When the Policy Engine is on, published rules govern this control; classic values saved through Edit anyway apply only after the engine is disabled. See What happens to classic settings.
- Classic settings
- Policy Engine
Turn it on for an app
Turn on each strategy that matches the traffic the app sends. Apps with any strategy on show a Token saving flag in the app list.
Going further with the Policy Engine
- Open Govern > Policy Engine > LLM Gateway > Controls > Token Savings.
- Start from a recipe, such as Turn on request compression or Clean up tool output, or use a section's add menu.
- Choose Applies to and the settings to change, then Save to draft.
- Use Test, then Review & publish to make the rules live.
Each switch resolves separately: highest priority wins, and On wins equal-priority ties. Off only exempts a scope when it outranks the matching On rule. Without a matching rule, request bodies are unchanged and requests go to the provider.
Use narrower scopes to enable only the transformations a pipeline needs. For example, compress structured retrieval results without rewording prose, or exempt one application at a higher priority.
Response cache
Response caching is separate from compression and from a provider's prompt cache. It returns an earlier answer for an eligible repeat of a non-streaming request, without a new provider call. A repeat can be exact, differ only in case, punctuation or filler words (Normalized), or be the same question in other words (Semantic). Exact and normalized repeats skip every guardrail; a semantic match is served only after request guardrails pass the new prompt. Use it only where reusing that answer is appropriate. See Response cache for match modes, safety checks and production results. Keep prompt cache warm instead maintains the provider's prompt cache; the model still answers the request.
Strategies
A transform only applies when it reduces token usage. Savings appear as Tokens saved in the app's Analytics tab and per request in Activity > Requests, and tenant-wide in Costs and savings.
Turn on only the methods that match your traffic. A retrieval app that sends JSON search results and Markdown snippets can enable Smart JSON compression and Markdown to text without enabling every transform.
Shared with the MCP Gateway
The same four methods, with the same names in settings, logs and analytics, are available for MCP tool output. On the MCP Gateway they rewrite tools/call text results before they return to the client, and OneMCP smart tool search also reduces tool-schema tokens. See MCP Gateway token saving.
For the Management API, the app's switches are (all off by default):
{
"smart_json_compression": false,
"html_to_text": false,
"markdown_to_text": false,
"text_compression": false
}
Related
- Standalone SDK token savings
- MCP Gateway token saving
- Response cache
- Conversation compaction - summarize the older part of long conversations
- Prompt store
- Costs and savings