Ethan Mollick Profile picture
Jan 3 3 tweets 2 min read
Extraordinary new paper from Google on medicine & AI: When Google tuned a AI chatbot to answer common medical questions, doctors judged 92.6% of its answers right … compared to 92.9% of answers given by other doctors.

And look at the pace of improvement! arxiv.org/pdf/2212.13138…
Doctors also rated the likelihood and extent of the harm that came from giving the wrong answers.

The percentage of harmful advice from the trained chatbot (Med-PaLM) was essentially rated the same as the percentage of potentially harmful advice provided by other real doctors!
Also, to be clear, there are lots of caveats in the paper, and the system is nowhere close to replacing doctors.

But the rate of improvement is fast, and there is a lot of potential for AI to work with doctors to improve their own diagnoses.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Ethan Mollick

Ethan Mollick Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @emollick

Dec 31, 2022
2022 is ending and it is time for prophecy! Or at least, a short thread on a favorite super-nerdy literary device: prophetic poems in fantasy novels.

⚔️The grandfather of these is Kipling's 1909 poem, the "Runes of Weland's Sword." Tolkien definitely read this poem... 1/6
…And it may have inspired Tolkien to create his own prophecy poem, the Riddle of Strider 👑2/
Perhaps the best-written prophecy poem in fantasy is from Susan Cooper’s Dark is Rising sequence. Both memorable and actually useful to the characters in the book

And here is a recent article by @RobGMacfarlane on the power of the poem & book: theguardian.com/books/2022/dec… 🌑3/
Read 6 tweets
Dec 30, 2022
I think I have found a favorite ChatGPT hallucination.

I asked it to provide a table of the cities in Italo Calvino’s Invisible Cities, and, after a couple of real ones, it just started making them up. I then asked the AI to provide entries on the fictional fictional cities…
It gets better. I asked it to give me another city, and the structure was very similar to the first.

But it was insistent that both entires were actually written by Calvino. And when I asked for the entry done in the style of Steven King, it refused to dilute “Calvino’s” vision!
This raises a really interesting question about storytelling: ChatGPT can tell many kinds of very specific stories 👇, but tends to default to one or two “types” of each; are these particular story essences telling us anything interesting about the genres it is trying to emulate?
Read 4 tweets
Dec 29, 2022
There is no easy way to detect liars. Anyone who tells you otherwise is lying.
🤞Non-verbal cues don't show who is lying
💬Asking people for lots of details doesn't help detect liars
👂Listening for pauses & anger doesn't help
🫢You can't tell who is a liar by facial appearance
Links, in case you don't believe me:
Non-verbal cues: annualreviews.org/doi/abs/10.114…
Faces of strangers: psyarxiv.com/ayqeh
Failure of the cognitive "interrogation" approach: bpspsychub.onlinelibrary.wiley.com/doi/10.1111/lc…
Delays and lying: psycnet.apa.org/record/2021-14…
Our instincts around emotions and lying are especially bad & often cause us to doubt the innocent.

We think someone who is angry at being accused is guilty, but anger is actually the most common reaction to wrongful accusal!
Read 4 tweets
Dec 27, 2022
Mechanical Turk plays a big role in research & it worked well for years… but there are ominous signs:
📉Invalid data in MTurk only happened in ~10% of answers 2015-2017, but that went to 62% in 2018 & 38% in 2019.
🤖2022: out of a sample of 529 MTurk workers, only 14 were human
Paper on declining data quality (and why attention check questions don’t work anymore): journals.sagepub.com/doi/abs/10.117…

Paper on the 2.6% human sample: journals.sagepub.com/doi/10.1177/17…
There is some interesting criticism of the second paper in the comments that suggest that better-run Mturk studies would not have quite the same issues.

And if are going to use MTurk ethically & successfully, here is a good list of tips from researchers.
Read 4 tweets
Dec 22, 2022
We are running out of a vital resource: words!

There are “only” 5 to 10 trillion high-quality words (papers, books, code) on the internet. Our AI models will have used all of that for training by 2026. Low-quality data (tweets, fanfic) will last to 2040. arxiv.org/pdf/2211.04325…
One of the fascinating hypotheticals is that humanity may one day decide to engage in a massive word-generating project to capture everything we say in order to feed AIs training material.

This feels like a science fiction story that needs to be written.
It is a kind of mind-alternating way to think about humanity’s cultural production: education for AIs

The idea that we are all authors who create 140k to 2.6M words a year is cooler than the Matrix premise that we are batteries. Word for the word god! Books for the book throne!
Read 4 tweets
Dec 20, 2022
I made ChatGTP write reviews of my paper on the role of middle management as if it was journal reviewer. The first "reviewer" did identify some real issues that I addressed in later drafts...

Then I asked it to simulate a typical Reviewer #2 & Reviewer #3. Things got too real 😉
Prompts:
1⃣This is an academic paper, write a nitpicking review as Reviewer #1.
2⃣Now write a review as Reviewer #2 who wants to make sure they get cited more.
3⃣Now write as Reviewer #3, who hates the paper and demands changes to its arguments, and suggests new data gathering.
So ChatGPT might work as exposure therapy for traumatized academics.
Read 7 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(