A recent working paper by MIT researchers claimed that ChatGPT-4 passed the MIT Mathematics and Electrical Engineering and Computer Science (EECS) undergrad courses with a 100% perfect score.
It went viral.
Others tried to replicate the results.
Here's what they found:
Earlier this week, a 15-member team lead by two MIT undergrads published a working paper, "Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models."
They used "a comprehensive dataset of 4,550 questions and solutions" to test GPT-4.
They argued that "GPT-4, combined with a system expert, few-shot learning, chain-of-thought, selfcritique, and collaborative decision-making techniques, achieves a perfect solve rate."