Research has shown demos give "cognitive legitimacy:" making a product plausible & inevitable. This paper on Shark Tank shows the more prepared (coherent, fact-based, narrative) the pitch, the more legitimate it seems & the more $ it raises! 1/4
Good entrepreneurs (as well as scam artists 👇) use demos and prototypes as part of an overall push for legitimacy. They are an important symbol of success for companies 2/4
Demos work in established companies, with caveats. If you are trying to pitch a novel idea to your boss, they will think you more creative & persuasive if you do a simple (crude) demo. If the idea isn’t novel, showing prototypes makes your boss think you are LESS creative. 3/4
We end with a famously effective demos. Watch how Jobs builds legitimacy: 1) Tells a story about why Apple’s win was inevitable 2) Demo'ed & described, the (secretly barely working) product 3) Combined personal style with careful preparation 4/4
• • •
Missing some Tweet in this thread? You can try to
force a refresh
As someone who has spent a lot of time thinking and building in AI education, and sees huge potential, I have been shown this headline a lot
I am sure Alpha School is doing interesting things, but there is no deployed AI tutor yet that drives up test scores like this implies.
I am not doubting their test results, but I would want to learn more about the role AI is playing, and what they mean by AI tutor, before attributing their success to AI as opposed to the other dials they are turning.
Google has been doing a lot of work on fine-tuning Gemini for learning, and you can see a good overview of the issues and approaches in their paper (which also tests some of our work on tutor prompts). arxiv.org/abs/2412.16429
I suspect that a lot of "AI training" in companies and schools has become obsolete in the last few months
As models get larger, the prompting tricks that used to be useful are no longer good; reasoners don't play well with Chain-of-Thought; hallucination rates have dropped, etc.
I think caution is warranted when teaching prompting approaches for individual use or if training is trying to define clear lines about tasks where AI is bad/good. Those areas are changing very rapidly.
None of this is the fault of trainers - I have taught my students how to do Chain-of-thought, etc. But we need to start to think about how to teach people to use AI in a world that is changing quite rapidly. Focusing on exploration and use, rather than a set of defined rules.
“GPT-4.5, Give me a secret history ala Borges. Tie together the steel at Scapa Flow, the return of Napoleon from exile, betamax versus VHS, and the fact that Kafka wanted his manuscripts burned. There should be deep meanings and connections”
“Make it better” a few times…
It should have integrated the scuttling of the High Seas Fleet better but it knocked the Betamax thing out of the park
🚨Our Generative AI Lab at Wharton is releasing its first Prompt Engineering Report, empirically testing prompting approaches. This time we find: 1) Prompting “tricks” like saying “please” do not help consistently or predictably 2) How you measure against benchmarks matters a lot
Using social science methodologies for measuring prompting results helped give us some useful insights, I think. Here’s the report, the first of hopefully many to come. papers.ssrn.com/sol3/papers.cf…
This is what complicates things. Making a polite request ("please") had huge positive effects in some cases and negative ones in others. Similarly being rude ("I order you") helped in some cases and not others.
There was no clear way to predict in advance which would work when.
The significance of Grok 3, outside of X drama, is that it is the first full model release that we definitely know is at least an order of magnitude larger than GPT-4 class models in training compute, so it will help us understand whether 1st scaling law (pre-training) holds up.
It is possible that Gemini 2.0 Pro is a RonnaFLOP* model, but we are only seeing the Pro version, not the full ultra.
* AI trained on 10^27 FLOPs of compute, an order of magnitude more than then GPT-4 level (I have been calling them Gen3 models because it is easier)
And I should also note that everyone now hides their FLOPs used for training (except for Meta) so things are not completely clear.