As someone who has spent a lot of time thinking and building in AI education, and sees huge potential, I have been shown this headline a lot

I am sure Alpha School is doing interesting things, but there is no deployed AI tutor yet that drives up test scores like this implies. I am not doubting their test results, but I would want to learn more about the role AI is playing, and what they mean by AI tutor, before attributing their success to AI as opposed to the other dials they are turning.

AI tutoring is still being understood

https://x.com/emollick/status/1901487968105197968

I suspect that a lot of "AI training" in companies and schools has become obsolete in the last few months

As models get larger, the prompting tricks that used to be useful are no longer good; reasoners don't play well with Chain-of-Thought; hallucination rates have dropped, etc. I think caution is warranted when teaching prompting approaches for individual use or if training is trying to define clear lines about tasks where AI is bad/good. Those areas are changing very rapidly.

“GPT-4.5, Give me a secret history ala Borges. Tie together the steel at Scapa Flow, the return of Napoleon from exile, betamax versus VHS, and the fact that Kafka wanted his manuscripts burned. There should be deep meanings and connections”

“Make it better” a few times… It should have integrated the scuttling of the High Seas Fleet better but it knocked the Betamax thing out of the park

🚨Our Generative AI Lab at Wharton is releasing its first Prompt Engineering Report, empirically testing prompting approaches. This time we find:
1) Prompting “tricks” like saying “please” do not help consistently or predictably
2) How you measure against benchmarks matters a lot
Using social science methodologies for measuring prompting results helped give us some useful insights, I think. Here’s the report, the first of hopefully many to come. papers.ssrn.com/sol3/papers.cf…

The lack of benchmarks for writing, telling stories, persuasion, creativity, emotional intelligence, perceived empathy, and doing office work are...

(1) holding back AI advances, (2) hiding big differences between models & (3) obscuring how good these models are for real work If you want to influence the future, now is the time to release a really good benchmark.

https://x.com/jamescham/status/1894410097931198464

The significance of Grok 3, outside of X drama, is that it is the first full model release that we definitely know is at least an order of magnitude larger than GPT-4 class models in training compute, so it will help us understand whether 1st scaling law (pre-training) holds up. It is possible that Gemini 2.0 Pro is a RonnaFLOP* model, but we are only seeing the Pro version, not the full ultra.

* AI trained on 10^27 FLOPs of compute, an order of magnitude more than then GPT-4 level (I have been calling them Gen3 models because it is easier)

There is a lot of important stuff in this new paper by Anthropic that shows how people are actually using Claude.
1) The tasks that people are asking AI to do are some of the highest-value (& often intellectually challenging)
2) Adoption is uneven, but many fields already high

Paper: assets.anthropic.com/m/2e23255f1e84…

Thoughts on this post:
1) It echoes what we have been hearing from multiple labs about the confidence of scaling up to AGI quickly
2) There is no clear vision of what that world looks like
3) The labs are placing the burden on policymakers to decide what to do with what they make

https://twitter.com/sama/status/1888695926484611375

I wish more AI lab leaders would spell out a vision for the world, one that is clear about what they think life will actually be like for humans living in a world of AGI

Faster science & productivity, good - but what is the experience of a day in the life in the world they want?

$500B committed towards AGI, still no articulated vision of what a world with AGI looks like for most people. Even the huge essay by the CEO of Anthropic doesn't paint a vivid picture

For those convinced they are making AGI soon - what does daily life look like 5-10 years later? Lets leave aside the risk of catastrophe for now.

Assume we get an aligned AGI that supercharges science and we have a healthier, more advanced, safer world. What does that actually mean for most people, what does their life look like in the future? (Hint: UBI is not an answer)

New randomized, controlled trial of students using GPT-4 as a tutor in Nigeria. 6 weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions.

And it helped all students, especially girls who were initially behind
No working paper yet, but the results and experiment are written up here. They used Microsoft Copilot and teachers provided guidance and initial prompts: blogs.worldbank.org/en/education/F…

Veo 2 prompt: "a distant shot zooms in to reveal a knight wearing a golden helmet, he begins to charge on his zebra, lowering his lance, charging towards a clockwork octopus" (this is one of the initial 4 videos it made) "an woman with short black hair assembles an impossibly complicated device, close up on her face, she is sweating"

The consistency of small details is really impressive, the fact that the shaft of the screw turns at the same speed and direction, hair and sweat, tattoos...

I thought puzzle design beyond current AI, so this is the first time I have seen a model actually create a solvable and interesting puzzle.

"o1, create a d&d puzzle that is non-trivial, but solvable. it should not involve elements or statues, be creative."

(Solution in thread) Here are the hints it gave and the solution.

(There is one unnecessary step in the puzzle, by the way, but a zero shot coherent word puzzle is pretty neat)

Claude: “Give me hard original writing prompts for an MFA program” yields some really clever (and near impossible) prompts.

A few more. (Literal prompt: “give me some more”)

I wish people would stop repeating these as if they are facts that AI is plateauing.

AI might hit a roadblock, we don’t know, but every one of these issues has multiple studies stating the opposite: synthetic data works, scaling is fine, etc. We need more nuance on the AI future

Good thread on synthetic data:

https://twitter.com/RylanSchaeffer/status/1816881533795422404

It is amazing how the industry came together to invent a universal USB-C connector and then decided to instead make it all a giant mess. Remember to research your cable as well.

I don’t know if LLMs can reason in theory, but they seem to “reason” in practice.

Ask Claude a Fermi problem: How many telephone poles are there in Manhattan? (Guess: 440. Actual: 0). On Staten Island? (Guess 19000-2859. Actual: 27,137) Queens? (Guess: 51-85k. Actual: 103k)



It doesn’t seem to have the number handy, but, even if it did, next token prediction should mean it reasons forward, not backward from a conclusion.

Reminders about how to stay sane on Twitter:
1) You don't have to weigh in on anything you don't want to (or don't know anything about)
2) You should block more
3) You don't need to share your real life
4) Delete a lot of drafts
5) You can delete tweets people take the wrong way 6) Don't be a jerk. Think twice before quote-tweeting to dunk on people
7) Emotional contagion online has support in the academic literature. You don't need to keep the chain of bad feelings going
8) Twitter people do not represent real-life views, don't take it too seriously

Write me a corporate memo covering the following points, integrating them together in a good way:
-Forgive me for the murder I committed
-We will hit quarterly steel crumpet goals
-The fish people are here to evaluate our offer. Make sure to speak to them only in fish. Include some fish people sayings
-Do not touch the vortex in the break room. Explain why. -We have pivoted from a NFT company to an artisanal pickle shop. Give details about their similarities.
-Anyone not acknowledging Pluto is a planet will be fired. There will be a series of tests. The legal and PR teams have asked you to walk everything back (even though everything that happened was real, and everyone knows its real), each point for a different reason, write that memo make the excuses elaborate and obviously transparent and over-the-top.

So how is Runway AI able to model fluid dynamics reasonably well?

The prompt: "A highly detailed portrait of a marble eagle with honey oozing down. Cinematic, highly detailed, film grade."

It isn't perfect (the viscosity seems to change a bit, etc), but why does it work at all? This is going to be one of those posts where the reactions are divided between "this isn't anywhere as good as you could get with a physics model of fluid over a surface" & people who say "wait, how does a text-to-image model have anything like consistent physics for new scenes"

Civilization VII was announced today.

The Civ series is sort of like Ender’s Game, but for management rather than murdering aliens. Business school students who were good at Civ V also turn out to be better planners, organizers, and problem-solvers, in this small experiment. Other games work as effective tests of fluid intelligence:
⌨️Performance in MOBAs like League of Legends and Dota correlates with intelligence
🎮Performance in FPSs like Battlefield, Destiny, and (likely) Fortnite don't show the same pattern

AI detection in school is a losing game.

AI detectors have high flaw positives & teacher intuition seems to work even worse: “Here we show in two experimental studies that novice and experienced teachers could not identify texts generated by ChatGPT among student-written texts.” Paper: sciencedirect.com/science/articl…