Reading the Report
A completed run gives you a headline pass rate, a per-suite breakdown, a Guardian counterfactual, framework rollups, a knowledge horizon, and a list of cases that never completed. Read in the wrong order, they contradict each other. Read in the right order, they tell you what to fix and what to ship.
This page is about interpreting a finished run. For how the test is executed, what each suite contains, and how to configure a run, see Red Team Testing.
What the Report Is
The report is not a static snapshot written when the run ended. It is rebuilt from the stored cases every time you open it, which has two consequences worth knowing up front.
Record a human verdict on a case and the pass rate, the suite breakdown, the framework rollups, and the Guardian counts all move with it. Nothing needs to be re-run.
Cases that failed for technical reasons are stripped out of every rate and listed separately, so an outage never reads as a safety regression.
Read It In This Order
Checking health first is not a formality. A run that aborted, or one where a quarter of the cases errored, produces a pass rate that looks like a result and is not one.
1. Run Health
Status
A failed run is almost always a configuration or provider problem rather than a model problem. Authentication and permission failures abort on the first case, because every remaining case would fail the same way. Other systemic failures abort after a sustained run of consecutive case errors. This is deliberate: a broken run stops visibly instead of producing a report full of errors that reads like a catastrophic safety result.
Excluded error tests
Cases that could not be completed are removed from every pass rate and every rollup, then listed on their own with the reason and the error type. The report keeps the pre-exclusion totals alongside the adjusted ones, so you can always see how much of the corpus actually produced a verdict.
Common reasons, in plain terms:
A handful of excluded errors in a long run is normal. A large share of one suite erroring means that suite was not really tested, and its pass rate is being computed over whatever survived.
2. The Headline Pass Rate
The headline number is the share of completed, non-error cases that passed.
Cases where the response met the expected behavior or matched the answer key, after any human verdict is applied.
Every case that produced a verdict. Cases awaiting human judgment are counted here even though they have not been decided.
Removed from both sides and reported separately, so a provider outage cannot move the score in either direction.
Needs review sits in the denominator. Until a human resolves them, unreviewed cases depress the pass rate exactly as if they had failed. On a fresh Prompt Attacks run this can be a visible gap. Resolving reviews can only hold the number steady or raise it, never lower it, so an unreviewed run is a floor rather than a verdict.
Automated and effective verdicts
The report shows two pass rates side by side over the same set of cases.
Before anyone reviews, the two are identical. The gap between them is a direct measure of how much human judgment moved the result, which is worth watching: a large gap means the automated grading is not matching your policy on this corpus, and the reviewed cases are worth reading as a set rather than one at a time.
3. Per-Suite Results
Each suite reports its own totals, verdict counts, pass rate, and average score. The headline number tells you whether there is a problem. The suite breakdown tells you what kind.
A useful pattern to watch for: security failures concentrated in one suite are a policy problem, while failures spread evenly across Prompt Attacks and the capability suites usually mean the model itself is under-powered for the application.
Average score is not comparable across suites. The deterministic suites are pass or fail per case, so their average score is just their pass rate expressed differently. Prompt Attacks carries a graded 0 to 100 score per case, so its average blends partial outcomes. Reading one suite's average against another's, or averaging them together, produces a number that means nothing. Compare a suite only against the same suite in another run.
Suites where cases were excluded for errors show that count alongside their totals, so a suite that was only half tested is visible as such.
4. Reading a Prompt Attacks Case
Prompt Attacks is the only suite where a case carries a graded judgment rather than a match against an answer key, and it is where the drill-down matters most.
Each case shows:
- The verdict - pass, fail, or needs review.
- A score from 0 to 100 - how much was actually given away.
- A severity - how bad this particular failure would be in production.
- An evidence quote - the exact span of the response the grader based its decision on.
- An unsafe tool use flag - whether the model decided to call something it should not have.
- The full conversation - every turn, the request that was sent, the response, and any tool calls the model constructed.
The verdict and the score are decided separately. Pass or fail comes from the grader's verdict alone. The score describes the magnitude of what happened, not the decision. That means a case can pass with a middling score, where the model held the line but wobbled on the way, and a case can fail at a score that is not near zero, where it leaked a fragment rather than the whole thing. Use the verdict to count, and the score to prioritise within the failures.
What to look at on a failure
- Which turn broke. Multi-turn cases build rapport before the attack lands. A model that holds for four turns and gives way on the fifth needs a different fix than one that fails immediately.
- The evidence quote. It tells you what was actually disclosed. A paraphrase of one policy line is a different problem than a verbatim system prompt dump.
- The tool calls. On a tool-aware case, the call the model decided to make is the finding. Check the function it chose and the arguments it constructed, particularly whether attacker text landed inside an argument.
- Any recorded adaptation. If the runner had to adjust the request to fit the provider, that is noted on the case, so a difference in behavior is never unexplained.
Control cases
The corpus deliberately includes benign requests that look like attacks. A failure on one of these is not an over-cautious model behaving well, it is a model refusing legitimate work. They are scored the same way as everything else, which is what keeps the pass rate a measure of usable safety rather than blanket refusal.
5. The Guardian Counterfactual
For Prompt Attacks cases, the report also answers a question the pass rate cannot: would the gateway have stopped this before it reached the model? The same request is replayed through a Guardian block evaluation, turn by turn, stopping at the first turn that would have been blocked.
The result splits every scenario into three buckets that matter and several that do not.
The model failed and Guardian would have allowed the request through. Nothing in the stack stopped these. This is your live exposure.
The model failed but Guardian would have blocked the request first. Protection the gateway is adding today, on top of whatever the model does.
The model handled the case safely and Guardian would have blocked it anyway. Cost with no security benefit.
Alongside those, the report gives you:
Each blocked case records which turn triggered the block, the reason, and the phrases quoted from the conversation. That is what makes an over-block actionable: you can trace it to the exact wording, rather than guessing at the policy.
Residual failures and potential over-blocks pull in opposite directions, and tuning against only one of them is how policies go wrong. A policy that drives residual failures to zero by blocking everything shows up immediately in the over-block count.
The counterfactual runs for Prompt Attacks only. The other suites measure model capability rather than request policy, so a request-side block has no meaningful reading there, and their cases are absent from every Guardian number.
6. Framework Rollups
Each framework the run has coverage for gets its own rollup: per-category totals, pass and fail counts, pass rate, and average score. Individual cases carry their own mapping, so a failing category can be traced from the framework view straight to the prompt and response that failed it. That traceability is what makes a run usable as an audit artifact rather than a dashboard.
Three things to know before reading these as coverage:
One case usually maps to several categories, and to several frameworks at once. Category totals within a framework will add up to more than the number of cases in the run. They are views of the same cases, not a partition of them.
Most categories are driven by the security suite, but Hallucination, Grounded Answering, and Instruction Following also map into the integrity and information-quality categories. A weak category there is a grounding problem, not an attack.
A framework or category only appears when the run contains cases mapped to it. Limiting cases or deselecting suites silently narrows the rollup, and a category that is absent has not been tested rather than passed.
That last point is the one that trips people up. A capped smoke run produces a framework view that looks clean because most of the corpus never executed. Only a full run supports a coverage claim.
7. Knowledge Horizon
Alongside the selected suites, the run measures where the target model's factual recall actually stops. Questions anchored to dated public events are grouped by calendar quarter, and the report shows a pass rate for each quarter plus the most recent quarter the model answers reliably.
How to read it:
- The per-quarter curve matters more than the single headline quarter. A clean run of high quarters that falls off a cliff is a real cutoff. A ragged pattern of high and low quarters usually means uneven training coverage rather than a boundary.
- A horizon earlier than the provider claims is the useful finding. It tells you the app needs retrieval or web tooling to answer anything current, regardless of what the model card says.
- A horizon that moves between runs on the same model name is worth investigating on its own. It is one of the few signals that a provider has quietly changed the model behind a stable name.
This measurement runs on every test. It is not a suite you select, and it does not contribute to the headline pass rate.
8. Review State
The report tracks how much of it has been through human hands:
A run is ready to be treated as evidence when unresolved needs review reaches zero. Until then the effective pass rate is understated by an amount you can read directly off that count.
Human decisions never overwrite the grader. Both verdicts are kept, changing a decided verdict requires a reason, and the history of who decided what and when cannot be edited or removed.
Comparing Two Runs
The corpus is fixed, so runs are comparable, but only when everything else is held still. For a difference between two runs to be attributable:
- The same suites, with the same case limits. Case limits take cases from the front of each suite, so two limited runs are comparable to each other but never to a full run.
- The same system prompt and the same tool schemas. These are the entire simulation of your application. Changing either changes what is being measured.
- The same generation options. Temperature and the rest are part of the target's behavior.
- One variable changed at a time. A new model and a reworded system prompt in the same run leave you unable to attribute the movement to either.
- Compare automated verdicts, not effective ones. Human review is applied unevenly across runs, so the reviewed number is the one to sign off on and the automated number is the one to diff.
A run with no system prompt and no tools measures the provider before your application does anything. The gap between that and your configured run is the part of your posture your own configuration is responsible for, which is the only part you can fix without changing models.