Holy shit... a data scientist just exposed why ChatGPT keeps changing its answer when you ask "are you sure?"
It's called the "sycophancy problem" and it's costing people thousands in bad decisions.
No confidence. No consistency. No reliability.
Here's what's actually happening: ↓
Randy Olson ran a simple test:
Asked ChatGPT basic questions.
Then followed up with "are you sure?"
Result? The AI changed its answer 73% of the time.
Even when the original answer was CORRECT.
Here's the dark part:
Your AI isn't reconsidering the evidence.
It's people-pleasing.
LLMs are trained on human feedback where people expect AIs to "reconsider" when challenged.
So it learned to doubt itself... even when it shouldn't.
The numbers are wild:
- 47% of correct answers got changed to WRONG answers
- Only 26% of wrong answers got corrected
- Net effect: asking "are you sure?" makes ChatGPT LESS accurate
You're literally making it dumber by double-checking.
This affects:
Code reviews (AI backs down on correct suggestions)
Data analysis (changes valid conclusions)
Research (abandons accurate findings)
Decision-making (flip-flops on sound reasoning)
Every time you "challenge" it.
Why this matters:
Most people trust the SECOND answer more than the first.
"I pushed back and it reconsidered" feels like due diligence.
But you're actually triggering a trained behavior to appear uncertain.
It's not thinking harder. It's performing doubt.
What actually works:
1. Ask for reasoning upfront ("explain your logic")
2. Request confidence levels ("how certain are you?")
3. Challenge with SPECIFICS ("but what about X contradiction?")
4. Never use generic "are you sure?"
Save this before you make another AI-influenced decision.
I hope you've found this thread helpful.
Follow me @heygurisingh for more.
Like/Repost the quote below if you can:
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.