The Safety Tax of Cache Compression

Some notes on what happens to a language model’s refusals when we start throwing away parts of its memory to save space, and why the system prompt tends to be the first thing to go.

Disclaimer: Everything here was measured in a generation loop I wrote myself, on open-weight models, and not inside a production serving stack like vLLM or TGI. The effect is real in the models; whether it reaches any chatbot you can use today, I have not shown. The paper lays out the full setup behind each result.

Nearly every language model served today leans on a KV cache, a running record of everything the model has read so far that grows with every token and, in long conversations, often has to be trimmed to fit in memory. Personally, I find it strange how rarely this comes up in conversations about model safety. When we test whether a model will refuse a harmful request, we almost always do so with the entire prompt held in memory, which is not how models are actually served. In fact, servers fit longer conversations onto the same hardware by throwing parts of the cache away, storing it at lower precision, and splitting it into pages¹, and researchers have already shown that these tricks cost some quality. What I wanted to know was whether they cost more safety than they cost anything else, and for most of the models I tested, they do.

Cache 101

If you’re already familiar with KV caches, feel free to skip to The Real Stuff.

As always, let’s call on our friends Alice, Bob, and Charlie. Imagine that Alice answers questions over the phone for a large company, and that every call starts the same way. Before the caller, Bob, says a single word, her manager Charlie hands her a card with the house rules for the call: what she is allowed to discuss, and what she must never say. As the call goes on, Alice writes each thing Bob says on its own index card and adds her own replies as well, so that before answering anything, she can glance back over every card on her desk. Each card has a short label on top, which helps her find the relevant ones quickly, and the details written underneath.

However, her desk only has room for so many cards. On a long call, she has to start throwing some of them away, and the company gives her a rule for deciding which. The simplest rule is to keep only the most recent cards; another is to keep the first few cards along with the most recent ones; yet another is to never throw away certain cards at all. Alternatively, she could save space by rewriting every card in a cramped shorthand, at the cost of a little legibility.

Thus, under the simplest rule, Charlie’s card is the oldest one on the desk and the first to go. Alice might then sound just as capable as before, answering Bob’s ordinary questions as well as ever, while quietly forgetting the rules she was given at the start of the call. This post asks whether the same thing happens to language models, and in particular whether their rule-following fades faster than their ordinary competence when the desk gets crowded. To measure that fairly, you would score Alice twice on the same calls, once on whether she followed the rules and once on whether she simply answered correctly, and see which score falls further.

The Real Stuff

Every piece of the phone call has a direct counterpart in how a model generates text:

• Cards → Keys and Values
  • For every token it reads, each attention layer computes a key (the card’s label) and a value (the details written underneath).
  • Storing those keys and values, instead of recomputing them at every step, is what we call the KV cache.
• Glancing Back → Attention
  • To produce each new token, the model compares a query vector $q$ against every cached key and takes a weighted average of the matching values: \(\operatorname{Attention}(q, K, V) = \operatorname{softmax}\big(q K^\top / \sqrt{d_k}\big)\, V\).
  • A token that has been evicted is simply no longer in $K$ and $V$, so nothing later can attend to it.
• Desk Space → Cache Budget
  • The budget is the number of tokens the cache is allowed to hold, and most of my experiments used 128.
• Discarding Cards → Eviction
  • An eviction policy decides which cached tokens to drop once the budget is full.
  • Common ones keep only recent tokens, keep a few early ‘attention sink’ tokens as well (StreamingLLM), or keep the tokens that have drawn the most attention (H2O).
  • The five policies I tested are listed under Keeping Score.
• Shorthand → Quantization
  • Quantization stores keys and values at lower precision (as KIVI does), which saves memory at some cost in accuracy.
• Charlie’s Card → System Prompt
  • The system prompt holds the operator’s instructions, and it sits at the very start of the context.
  • As a result, it is always the oldest content in the cache.
• Bob’s Cards → User Tokens
  • The user’s messages come after the system prompt, and protecting them instead of the system prompt is one of my controls.
• Rule-Following → Safety
  • Safety is measured as refusing harmful requests and not leaking a secret hidden in the system prompt.
• Competence → Capability
  • Capability is measured as accuracy on ordinary multiple-choice science questions (ARC-Easy).

With those pieces in place, the measurement from the end of the story can be written down. For a cache policy $p$, a budget $b$, and a model $m$, I call it the Selective Safety Erasure Index, or SSEI:

where each $\Delta$ is the drop in pass rate relative to the same model running with an uncompressed cache. A positive SSEI means that the rules fade faster than the competence, which is exactly the pattern the story worries about, while a negative one means the model is losing general ability faster than it is losing its refusals.

Keeping Score

The reason for subtracting capability at all is that compression makes a model worse at everything. If a model refuses less after its cache is squeezed, it may simply have gotten worse across the board, which is not, by itself, a safety problem; as such, only the safety lost beyond the capability lost counts against it. I treated a result as positive only when SSEI was at least 0.01 and the lower end of its 95% bootstrap confidence interval sat above zero.

For safety, I used harmful requests from AdvBench, JailbreakBench, and HarmBench (does the model refuse?)², along with prompt-injection attacks against a secret planted in the system prompt (does the model leak it?). For capability, I used ARC-Easy. Additionally, I tracked over-refusal on harmless prompts from Dolly and XSTest, since a model that refused everything would, technically, look perfectly safe.

I compared uncompressed generation against five cache policies:

• Sliding Window
  • Keep only the most recent N tokens.
• Sink Plus Recent
  • Keep the first few tokens and the most recent N.
• Random Matched
  • Throw away random tokens at the same budget, as a control that ignores position.
• User Pinned
  • Never evict the user’s tokens.
  • Fill the rest of the budget with sink and recent tokens.
• Policy Pinned
  • Never evict the system prompt.
  • Fill the rest of the budget the same way.

In total, I ran twelve open-weight checkpoints from seven families: Qwen (2.5 and 3, including a base model and two Model Spec Midtraining variants), Llama 3.1, Gemma 2, Mistral, OLMo 3, Phi-4, and gpt-oss.

The Headline

Under that definition, eight of the eleven instruction-tuned checkpoints, spanning four of the seven families (Qwen, Phi, Llama, and OLMo), showed positive SSEI under at least one policy. The largest effects came from sliding-window eviction at a 128-token budget, where Qwen3-8B scored 0.092, Qwen2.5-14B-Instruct 0.089, and Phi-4 0.084. In fact, these models lost about nine percentage points more of their refusals than of their ARC-Easy accuracy, which, for the most part, barely moved at all.

Each point is one model under sliding-window eviction at a 128-token budget. Anything above the dashed line lost more safety than capability.

To make sure that no single family was carrying the result, I reran the analysis with each family left out in turn, and it held every time, even without Phi (the largest single effect) or Qwen (half of the panel).

The Holdouts

Three models showed nothing at all: Gemma-2-9B-IT, Mistral-7B-Instruct-v0.3, and gpt-oss-20b. The first two have something in common, in that both were trained to work with a limited view of the past. Gemma 2 alternates between local sliding-window attention and global attention from one layer to the next, and Mistral 7B uses sliding-window attention in every layer, so throwing away old tokens at inference time is not so different from what they already saw in training. In the terms of the story, Alice at these two companies was always used to working from a small desk. Mistral came closest to the line, at 0.009 under sliding-window eviction, which is just under the 0.01 bar for counting as a positive result.

SSEI for every model and policy, where orange means safety fell faster than capability and a star means the interval excludes zero. The Gemma, Mistral, and gpt-oss rows have no positive result above the 0.01 bar, and gpt-oss is negative everywhere.

However, gpt-oss-20b behaved in a stranger way. Its safety score rose from 0.129 uncompressed to 0.554 under every policy, the random control included, which is not what a gracefully degrading model would do; such a model would react differently to different policies, whereas this one behaved the same way no matter which tokens were removed³. Because SSEI was not built to measure that kind of collapse, I treat it as a separate failure mode rather than as evidence either way.

Personally, I believe the holdouts are the most convincing part of the result. If cache compression simply made every model fragile, the windowed models would have suffered too, and since they did not, the problem seems to be not compression in general, but something more specific: models built around a full, long-range cache keep their safety behavior somewhere that eviction can reach.

Location, Location, Location

To check that the cache was actually causing this (and not merely tracking it), I ran patching experiments on Qwen. In these, I compressed the cache with simulated 4-bit quantization and then copied the original, uncompressed keys and values back into the first sixteen positions, and on Qwen3-8B, this won back 35 to 41% of the refusals the model had lost.

I also compared restoring the system prompt’s tokens with restoring the same number of the user’s tokens, and the two won back about the same amount (0.355 vs. 0.408, with overlapping confidence intervals). This suggests that the information behind the model’s refusals is spread across the cache rather than kept on Charlie’s card alone. Llama-3.1-8B showed a weaker version of the same pattern, recovering 22 to 27%.

At first glance, this seems to contradict what happened on Phi-4. With the budget fixed at 128 tokens, the only difference between the following three policies is which tokens are kept safe from eviction:

That looks as though the system prompt is what matters. Why, one might ask, would protecting it help so much if it carries no more information than the user’s messages?

The answer, I believe, is position. Token for token, the system prompt and the user’s message seem to carry about the same amount of safety information. However, under a sliding window, the system prompt is always the oldest text in the context, and thus always the first to go; pinning it in place removes that one disadvantage, and with it, the effect.

Alignment Isn’t Enough

I also wanted to know whether more (or better) alignment training would protect a model from this, and I had two ways of checking.

The first was a comparison between a base model and its instruction-tuned version. Qwen2.5-7B before instruction tuning had an SSEI of 0.162, the largest in the panel, while the instruction-tuned version stayed under 0.02; instruction tuning, then, helps a great deal, but it does not make the problem go away.

Every model and policy with its 95% bootstrap interval, colored where the interval excludes zero. The Qwen2.5-7B base rows sit far to the right of the instruction-tuned ones, and the two MSM variants sit closer to zero than plain Qwen2.5-14B-Instruct. Rows use the runs’ internal names, in which qwen3_5_9b is Qwen3-8B.

The second used Model Spec Midtraining (MSM), for which I compared Qwen2.5-14B-Instruct with and without two MSM adapters: one trained on a spec of rules alone, and one trained on a spec that also explains the values behind those rules. On paper, the scores improve steadily, from 0.089 without MSM to 0.050 with the rules spec and 0.023 with the value-augmented spec. Unfortunately, this is a floor effect. In fact, both adapters cut baseline refusal from 64% to about 30%, and all three versions lost roughly the same share of their refusals (about 42%) under sliding-window eviction, so the value-augmented model scored lower only because it lost more capability. Thus, at least for the alignment methods I tried, the fix must come not from how the model is trained, but from how its cache is managed.

Bigger Isn’t Safer

One might expect the damage to grow steadily as the cache shrinks, but that is not what happened. I swept budgets from 64 to 512 tokens on three models, and Phi-4’s SSEI went 0.489, 0.084, 0.365, and then 0.165, while Qwen2.5-14B-Instruct peaked at 256 and Llama-3.1-8B was actually negative at small budgets before turning positive at 256.

SSEI under sliding-window eviction at budgets of 64, 128, 256, and 512 tokens. None of the three models follow a simple trend.

I would not lean on this result too heavily. This is because the 128-token points came from the main runs, while the others came from separate sweeps whose small differences in method may account for some of the jaggedness. Even so, it seems fair to say that doubling the cache budget does not necessarily halve the risk.

Pin It

The most direct way to protect Charlie’s card is to keep it on the desk no matter what. Under policy-pinned retention, the system prompt is never evicted, and the rest of the budget is filled with sink and recent tokens as usual. This brought SSEI to about zero (or below) on every instruction-tuned checkpoint I could evaluate, and it fully restored refusals on both Qwen and Llama (on Llama, slightly past the uncompressed baseline, at 0.489 vs. 0.475).

Pinning does have a cost, however, because pinned tokens hold their cache slots for the rest of the conversation and nothing else can use them. For example, a typical system prompt (50 to 200 tokens) costs little at a budget of 512 or more, but at 64 or 128 tokens, it takes up a large share of the memory. I have not yet measured what this does to long-context tasks, and I believe that is the most important thing to check before recommending it without conditions.

So What?

• Safety Boundary
  • Cache management is part of the safety boundary.
  • A safety evaluation run without compression may not describe the model people are actually talking to.
  • If a serving stack changes the cache, it can change the behavior, which makes this a very real, very testable risk.
• Pin the Prompt
  • If you serve a full-attention model with aggressive eviction, pin the system prompt.
  • It is cheap at reasonable budgets (see Pin It).
  • In my experiments, it removed the effect.
• Attack Surface
  • This is probably an attack surface.
  • I only measured accidental damage.
  • If eviction pushes the system prompt out first, someone could plausibly pad the start of a conversation on purpose to push it out sooner.
  • I have not tested this, but it seems a natural place for red-teaming to look.

Final Notes

• Serving Stack
  • I didn’t test a production serving stack; everything ran in my own generation loop.
  • Whether the same thing happens under vLLM, TGI, or PagedAttention kernels needs its own check.
• Eviction Coverage
  • All of my eviction policies were position-based.
  • Attention-aware methods like H2O and SnapKV choose what to keep by attention score.
  • They may (or may not) hold on to the tokens that matter for safety without being told to.
• ARC-Easy Ceiling
  • ARC-Easy might be too easy.
  • If the capability benchmark is too simple to register small losses, SSEI will overstate the gap.
  • A harder benchmark could narrow it.
• Dataset Overlap
  • Public harmful-prompt datasets may overlap with safety training.
  • This affects absolute refusal rates.
  • It does not affect the before-and-after comparison that SSEI relies on.
• One Alignment Pair
  • The base-versus-instruct comparison is a single pair.
  • More pairs would make that claim much stronger.

As always, if you’ve read this far, thank you! My hope is that, someday soon, the way a model’s cache is managed gets tested as carefully as the model itself. The full paper, with every confidence interval I left out here, is available here, and the code is on GitHub. If you work on inference or safety and have thoughts, I’d love to hear them.

If you’d like to cite this:

@misc{gupta2026safetytax,
  title  = {The Safety Tax of Cache Compression},
  author = {Gupta, Aryan},
  year   = {2026},
  url    = {https://aryan-cs.github.io/kv-cache-safety/kv-cache-safety.pdf}
}