Thread by @deedydas

o1-preview is far superior to doctors on reasoning tasks and it's not even close, according to OpenAI's latest paper.

AI does ~80% vs ~30% on the 143 hard NEJM CPC diagnoses.

It's dangerous now to trust your doctor and NOT consult an AI model.

Here are some actual tasks:

1/5

Here's an example case looking at phosphate wasting and elevated FGF23, then proceeded to imaging to localize a potential tumor.

o1-preview suggested testing plan takes a broader, more methodical approach, systematically ruling out other causes of hypophosphatemia.

2/5

For persistent, unexplained hyperammonemia, o1-preview recommends a prioritized expansion of tests—from basic immunoglobulins and electrolytes to advanced imaging, breath tests for SIBO and specialized GI biopsies—ensuring more common causes are checked first.

3/5

I have all the respect in the world for doctors, but in many cases their job is basic reasoning over a tremendously large domain-specific knowledge base.

Fortunately, this is exactly what LLMs are really good for.

This means more high quality healthcare for everyone.

4/5

Source:

5/5arxiv.org/pdf/2412.10849