The Spectral Geometry of Misalignment
A tour of what post-training leaves behind in a model’s weights, and why the directions that matter for refusal and misalignment are far easier to find than to isolate.
A tour of what post-training leaves behind in a model’s weights, and why the directions that matter for refusal and misalignment are far easier to find than to isolate.
An account of what happens when we steer a language model toward honesty, and whether it actually becomes more truthful or simply learns to sound that way.
Some notes on what happens to a language model’s refusals when we start throwing away parts of its memory to save space, and why the system prompt tends to be the first thing to go.
What happens when a model reads a company’s SEC filing before predicting how its stock will react, and why the text turns out to say more about risk than about returns.
Some notes on what happens to poker when the deck shrinks to a handful of numbered cards, and why the only stable way to play even three of them involves a little bluffing.