Thread by @matei_zaharia

We benchmarked coding agents on our own internal tasks at Databricks and learned a lot!

There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now. 🧵

Why did we build an internal coding benchmark? Public benchmarks like SWE-Bench are often over-tuned for, so we took real tasks our engineers did and curated test suites for them to see which agents can solve them end-to-end. databricks.com/blog/benchmark…

These are the results on OUR sample of OUR codebase, so they are not meant to be comprehensive, but we think many companies can do a similar internal benchmark. Several findings immediately stood out:

1) Many models are now competitive at the top tier, including open source.

2) GLM 5.2 in particular was a major step forward in open source coding agent performance, even in our own codebase that is VERY different from SWE-Bench and TerminalBench (lots of Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet, etc).

3) Harnesses make a huge difference in cost-performance. The very simple Pi harness (@badlogicgames) got the same success rate as harnesses from the LLM vendors with Opus and GPT 5.5, but at 2x less cost! Seems to be mainly due to smaller inputs to the LLM.

4) Cheaper per-token does not imply cheaper per-task. For example, Sonnet 5 costs less per token than Opus 4.8 but used more tokens, resulting in higher cost and lower quality. Interestingly my ex-student @ChenLingjiao had also found this on other tasks: arxiv.org/abs/2603.23971

Read more in the blog how we built the benchmark and what we're doing with the findings. This is partly why we built @omnigent_ai as a "meta-harness" to let developers switch and compose agents, and Unity AI Gateway to analyze and gate LLM usage centrally. databricks.com/blog/benchmark…