Thread by @egrefen

🌶️(?) take: Agents are somehow hot right because people realized that LLM output can be interpreted as a DSL which directs side effects in the world (e.g. tool calls) rather than just returning text in a chat/autocomplete sense. What are the open challenges? A 🧵... [1/11]

Setting aside the fact that people are literally re-inventing a bunch of terminology around agents when we have several decades of framing from RL (POMDPs, anyone?) that we can rely on to describe what we're doing, the main challenges are as follows, working backwards. [2/11]

1. Can we determine what capabilities and downstream use cases we care about? This, as a whole, is a fairly open-ended problem, and the use cases etc can individually be open-ended/underspecified, but having some notion of what we're building towards is a good start. [3/11]

1. (continued) You'd be surprised about how many people are into bUiLdInG aGeNtS without thinking precisely about this, which is why a lot of agent building startups will burn through VC cash trying to find PMF and fail. [4/11]

2. If we know what to build, can we define and measure success? We completely lack evaluation measures for complex agentic behavior, which limits what we can develop agentic behavior towards. [5/11]

3. If you know how to evaluate agents, can we produce semi-passable automatic evaluations? Can we ensure these stay roughly aligned with human judgements (or other measures of extrinsic utility) as use behaviors shift? Can we run these cheaply and at scale? [6/11]

4. If we have automatic evaluations, can we generate/acquire data to run agents over in a scalable manner, in order to automate the improvement loop? [7/11]

4. (continued) Can we ensure that such data is a good distributional proxy for data observed at deployment time, or at least covers that distribution? Approaches like self-play/unsupervised environment design can be explored here, in addition to varieties of RL(HF/AIF/etc) [8/11]

If the answer to all the above is yes, then we're in a good position to build better agents, and what that entails will fall on a spectrum between "Everything works with a little prompt engineering, RL, and distillation" to "Nothing works and we need to do novel science". [9/11]

For commercial purposes, we hope for the former, but as researchers we relish the latter. Unfortunately, the number of people wanting to invest time in building the top of this pipeline is much lower than the number who want to harvest low hanging fruit at the bottom. [10/11]

As a closing note, I'm always surprised at how it's only popped up as a hot button topic in the last year or so (for LLMs, not RL). I thought it was the entire point of the LaMDA tech report from early 2022. Three years is a long time in ML these days. [11/11]