AI has the potential to transform real-world domains. But can AI actually improve outcomes in live interactions?

We conducted the first large-scale intervention of a Human-AI Approach that has statistically significant positive learning gains w/ 900 tutors & 1,800 K12 students. Link: arxiv.org/pdf/2410.03017

Students from underserved communities have the most to gain from high-quality education.

But, training teachers in real time is $ and hard to scale.

LLMs also have major limitations.
1) LLMs are trained on Web data: They have never seen real kid behavior or how good teachers think as they teach.
2) LLMs feel disconnected from real-world settings. You constantly need to switch to ChatGPT/Claude's interface which disrupts the flow of live interactions.

We talk a lot about the potential of AI for applications, like AI for Education.

But actual progress requires that we hill-climb on realistic, hard tasks. Are there any?

🔽 Bridge, Backtracing, and Teacher Coach are 3 real-world AI for Education datasets that are far from saturated! [1] Can LMs help K-12 students with math mistakes? It's NOT enough to just reveal the right answer to them!

🌉 Bridge:

Input: Real conversation with student
Output: Response to student mistakes
Metric: Pairwise comparison w/ expert teacher response huggingface.co/datasets/rose-…

How can we algorithmically figure out what our model doesn’t know, and then construct datasets to improve it?
We tackle this question in “Know thy student: Interactive learning with Gaussian processes” at #ICLR2022 @cells2societies workshop.
Paper: openreview.net/pdf?id=rpGGNrM…
[1/N] We cast this problem as a teacher-student setup where the teacher must first interact to diagnose 🧪the student (the model), before teaching 👩‍🏫(constructing the training dataset).
[2/N]