We benchmarked coding agents on our own internal tasks at Databricks and learned a lot!

There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now. 🧵 Why did we build an internal coding benchmark? Public benchmarks like SWE-Bench are often over-tuned for, so we took real tasks our engineers did and curated test suites for them to see which agents can solve them end-to-end. databricks.com/blog/benchmark…

Really excited to open source a new project: Omnigent, a meta-harness for AI agents.

It lets you build multi-agent coding and custom agents, sitting above Claude Code, Codex, Pi, and agent SDKs to let you compose them. It also adds live collaboration and rich control policies. Omnigent is based on the trend we saw with AI usage at Databricks and Neon: engineers were combining multiple agents into loops and workflows, and this was difficult above the harness layer. We add a uniform API above any harness that enables rich features on top.

Really cool result from the Databricks research team: You can tune LLMs for a task *without data labels*, using test-time compute and RL, and outperform supervised fine-tuning! Our new TAO method scales with compute to produce fast, high-quality models. databricks.com/blog/tao-using… Our new method, Test-time Adaptive Optimization (TAO), only needs input examples of a task and can outperform supervised fine-tuning on thousands of human-labeled examples. It brings OSS models like Llama to the quality of expensive larger models.