Skip to main content

Token saving

Updated on 3 November 2026.

Reduce tokens sent to the provider with request compression and tool-output cleanup. The Policy Engine can also reuse an eligible cached response. Your application keeps using its existing API.

For optimization from your own application code, use standalone SDK token savings. It accepts a text string and an explicit choice of seven methods, returns the final text and savings estimates, and supports deterministic, cache-friendly prompt preparation without routing an LLM call through the gateway.

Token saving on the AI gatewaysExplainer1:58

How the LLM Gateway and MCP Gateway rewrite JSON, HTML, Markdown and verbose text into fewer tokens before it reaches the model.

YouTube All videos

Configure this control​

Choose the mode used by your gateway. When the Policy Engine is on, published rules govern this control; classic values saved through Edit anyway apply only after the engine is disabled. See What happens to classic settings.

Going further with the Policy Engine​

  1. Open Govern > Policy Engine > LLM Gateway > Controls > Token Savings.
  2. Start from a recipe, such as Turn on request compression or Clean up tool output, or use a section's add menu.
  3. Choose Applies to and the settings to change, then Save to draft.
  4. Use Test, then Review & publish to make the rules live.
SectionWhat it changes
Smart JSON compressionCompress eligible JSON before it reaches the provider.
HTML to textKeep text from HTML in requests.
Markdown to textRemove Markdown formatting in requests.
Text compressionShorten prose before it is sent. This can reword text.
Minify JSON tool outputsRemove whitespace from JSON tool outputs.
Clean up tool outputRemove color codes, progress bars and repeated lines.
Remove encoded contentReplace long base64 content in tool outputs with a size note; the model no longer receives that content.
Response cacheAnswer eligible repeat requests from a stored response rather than sending every request to the provider. Choose Exact, Normalized or Semantic matching.

Each switch resolves separately: highest priority wins, and On wins equal-priority ties. Off only exempts a scope when it outranks the matching On rule. Without a matching rule, request bodies are unchanged and requests go to the provider.

Use narrower scopes to enable only the transformations a pipeline needs. For example, compress structured retrieval results without rewording prose, or exempt one application at a higher priority.

Response cache​

Response caching is separate from compression and from a provider's prompt cache. It returns an earlier answer for an eligible repeat of a non-streaming request, without a new provider call. A repeat can be exact, differ only in case, punctuation or filler words (Normalized), or be the same question in other words (Semantic). Exact and normalized repeats skip every guardrail; a semantic match is served only after request guardrails pass the new prompt. Use it only where reusing that answer is appropriate. See Response cache for match modes, safety checks and production results. Keep prompt cache warm instead maintains the provider's prompt cache; the model still answers the request.

Strategies​

StrategyWhat it doesBeforeAfter
Smart JSON compressionCompacts eligible JSON objects and arrays, converting to TOON where that saves more. Best for tool results and structured data. Up to about 20% savings.{"name": "John", "age": 30}name:John|age:30
HTML to textStrips HTML markup down to readable text.<p><b>Hello</b> world</p>Hello world
Markdown to textStrips Markdown syntax that costs tokens without adding meaning.## Hello **world**Hello world
Text compressionRemoves low-value prose and separator noise from long text while keeping its meaning. Structured-looking lines are left alone.Please review the following statement and the context which was actually very repetitive.Review statement and context.

A transform only applies when it reduces token usage. Savings appear as Tokens saved in the app's Analytics tab and per request in Activity > Requests, and tenant-wide in Costs and savings.

Turn on only the methods that match your traffic. A retrieval app that sends JSON search results and Markdown snippets can enable Smart JSON compression and Markdown to text without enabling every transform.

Shared with the MCP Gateway​

The same four methods, with the same names in settings, logs and analytics, are available for MCP tool output. On the MCP Gateway they rewrite tools/call text results before they return to the client, and OneMCP smart tool search also reduces tool-schema tokens. See MCP Gateway token saving.

Setting keyUse it for
smart_json_compressionStructured JSON payloads, API responses and tool results.
html_to_textScraped pages, rich emails, dashboards and HTML reports.
markdown_to_textMarkdown documents, README files, issue bodies and generated notes.
text_compressionVerbose prose, repeated separator lines and low-signal plain text.

For the Management API, the app's switches are (all off by default):

{
"smart_json_compression": false,
"html_to_text": false,
"markdown_to_text": false,
"text_compression": false
}