Would an LLM tell you if it’s gaming your eval? Often, no. But we can still catch the model thinking about it.
New research: we measure how close a model comes to saying it’s being tested. This detects eval awareness with 10× to 100× fewer samples than monitoring model outputs.🧵
New research: we measure how close a model comes to saying it’s being tested. This detects eval awareness with 10× to 100× fewer samples than monitoring model outputs.🧵
The idea is simple: we estimate how likely the model is to say it's being tested (throughout the chain of thought), by reading directly from its next-token logits.
If this probability really captures eval awareness, it should spike on prompts from evaluations. It does: monitoring this probability using a single rollout separates evaluation from real-user prompts better than an LLM judge reading 100 rollouts.
This works even if we monitor before the model starts writing any CoT. Also, the probabilities depend little on whether the model says it's being tested. So: the model claiming to be tested could be more a reflex the prompt sets off than a behavior emerging over the CoT.
In prior work with @AISecurityInst we found verbalized eval awareness can inflate how safe a model appears when evaluated. This motivated me to develop better tools for catching eval awareness.
Monitoring probabilities is a cheap, interpretable way to detect eval awareness: a richer signal than reading the model's outputs, and one that may be easier to interpret than activation probes.
Full post: lesswrong.com/posts/PK7ZvFZx…
Full post: lesswrong.com/posts/PK7ZvFZx…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
