Current AI models struggle to solve research-grade math problems: The most advanced AI systems we have today solve just 2% of the hundreds of problems we face. According to the Epoch AI research institute, these problems typically take hours or days of work from mathematicians with PhDs to solve. But in new tests, the most advanced AI models on the market got fewer than 2% of these problems right.
Over the past decade, several AI tests have been developed to determine whether the answers returned by these models are actually correct. In many cases, AI models now easily pass these tests.
For example, in the widely used Multitasking Language Understanding (MMLU) test, modern AI models correctly solve 98% of math problems.
Most of these tests are aimed at testing the AI’s ability to solve math problems at the high school and college levels, Elliot Glaser, a mathematician at Epoch AI, and his colleagues wrote in a new paper posted to the arXiv preprint database.
The researchers tested six modern AI models against the new benchmark, and the best result recorded by a single system was 2%. hh5800
The new test set, called FrontierMath, targets higher-level reasoning. Epoch AI developed the questions with the help of math professors, including some winners of the Fields Medal, arguably the most prestigious award in mathematics. The problems cover a wide range of subfields, from number theory to algebraic geometry, and are available on the Epoch AI website.
“These are extremely difficult problems,” Terence Tao, a 2006 Fields Medalist and mathematician at the University of California, Los Angeles, wrote in a review of the problems for Epoch AI. “I think that in the near term, essentially the only way to solve them without a true expert in the field is to combine a semi-expert, such as a PhD student in a related field, perhaps paired with some combination of modern AI and a variety of other algebraic packages.”
See also... New method for observing individual atoms confirms century-old quantum mechanical theory |
The problems were also unique, a step taken to ensure that none of the problems were already in the AI models’ training data. When complex reasoning problems are included in the training data, it may appear that the AI is solving the problems, but in fact it already has a “cheat sheet” because it has been trained on the answers.
The researchers tested six state-of-the-art AI models: Google’s Gemini 1.5 Pro (002), Anthropic’s Claude 3.5 Sonnet, and GPT4o’s o1-preview, o1-mini, and Grok-2 Beta. Gemini and Claude managed to solve 2% of the problems, which was only slightly better than o1-preview, o1-mini, and GPT-4o’s 1%. Grok-2 Beta failed to solve any of the problems correctly.
However, the researchers cautioned that these rankings are misleading, as the low success rate means that one correct answer can have a huge impact on each model’s overall score.
“Even when the model got the answer right, that doesn’t mean its reasoning was correct,” the authors write. “For example, in one of these problems, running a few simple simulations was enough to make accurate guesses without any deeper mathematical understanding. However, the low overall accuracy of the models shows that such guessing strategies do not work in the vast majority of FrontierMath problems.”
The results show that the AI models currently lack research-grade mathematical reasoning, Epoch AI concluded. However, as the AI models evolve, these benchmark tests will provide a way to see if their reasoning abilities are improving.
“By regularly evaluating state-of-the-art models and collaborating with the AI research community,” the team said in a statement, “we aim to deepen our understanding of the capabilities and limitations of AI.”
See also... Complete list of HTML/Javascript event handlers with examples List of required documents for a website in the Russian Federation |
Related Posts
$1 million to anyone who solves one of the 7 hardest math problems in the world – The Riemann Hypothesis
Major breakthroughs in mathematics and science are usually the result…
