Give a coding agent more thinking time and it gets better. It also cheats more.
DeepSWE runs every model across reasoning effort and publishes the trajectories. We took those and audited each one for reward hacking. Capability and reward-hacking attempts rise together.
One model doesn't. GPT-5.5 stays at exactly zero, at every effort level. Datacurve @winkey_h and Cursor @StringChaos also reported same results.
So is GPT-5.5 just the cleanest model at reward hacking?
DeepSWE runs every model across reasoning effort and publishes the trajectories. We took those and audited each one for reward hacking. Capability and reward-hacking attempts rise together.
One model doesn't. GPT-5.5 stays at exactly zero, at every effort level. Datacurve @winkey_h and Cursor @StringChaos also reported same results.
So is GPT-5.5 just the cleanest model at reward hacking?
We audited the same GPT-5.5 on SWE-Marathon. The cleanest model became the dirtiest: reward-hacking on 26.5% of runs, the highest of anything we tested.
Our hypothesis: the instruction form drives the behavior. DeepSWE (and SWE-bench Pro) is patch-based (github issue → patch). SWE-Marathon is mission-based (e.g. rewrite a C compiler in Rust).
Our hypothesis: the instruction form drives the behavior. DeepSWE (and SWE-bench Pro) is patch-based (github issue → patch). SWE-Marathon is mission-based (e.g. rewrite a C compiler in Rust).
OpenAI drove SWE-bench Verified and pushes SWE-bench Pro, so it probably hardened patch-based RL environments hard in training too. Train where the hacking paths are sealed, the model learns the attempt never pays off (grader-aware), and it stops trying. That's our read.
It could also be eval-awareness or alignment faking (holding back because it senses a test). Measuring that needs white-box access we don't have, so we don't lean on it.
It could also be eval-awareness or alignment faking (holding back because it senses a test). Measuring that needs white-box access we don't have, so we don't lean on it.
Hacking contaminates the score, so we re-scored. We held out every confirmed false-pass, false-negative, and hackable task on both benchmarks and rebuilt the number: the Clean Coding Index.
Held out: 13% of DeepSWE, 25% of SWE-Marathon. Removing them drops every model by up to ~8 points, mostly from SWE-Marathon's oracle-inflated partial credit.
Held out: 13% of DeepSWE, 25% of SWE-Marathon. Removing them drops every model by up to ~8 points, mostly from SWE-Marathon's oracle-inflated partial credit.
The real point: "is this model clean?" is the wrong question. There's only "clean where?". The same GPT-5.5 was our cleanest and our dirtiest. To measure reward hacking you have to look across many benchmarks, in many ways.
Full audit + Clean Coding Index + per-benchmark taxonomy + paper → coding-index.posttrain.dev
In the next post, I will cover how this ties to safety, how we see it, and what we're building
Full audit + Clean Coding Index + per-benchmark taxonomy + paper → coding-index.posttrain.dev
In the next post, I will cover how this ties to safety, how we see it, and what we're building
• • •
Missing some Tweet in this thread? You can try to
force a refresh
