{"id":1323,"date":"2025-04-30T10:30:10","date_gmt":"2025-04-30T07:30:10","guid":{"rendered":"https:\/\/planeta21vek.ru\/artemiada\/eng\/?p=1323"},"modified":"2025-11-20T19:37:14","modified_gmt":"2025-11-20T16:37:14","slug":"mathematicians-have-stumped-the-most-advanced-models-of-generative-artificial-intelligence","status":"publish","type":"post","link":"https:\/\/planeta21vek.ru\/artemiada\/eng\/2025\/04\/30\/mathematicians-have-stumped-the-most-advanced-models-of-generative-artificial-intelligence\/","title":{"rendered":"Mathematicians have stumped the most advanced models of generative artificial intelligence"},"content":{"rendered":"

Current AI models struggle to solve research-grade math problems: The most advanced AI systems we have today solve just 2% of the hundreds of problems we face. According to the Epoch AI research institute, these problems typically take hours or days of work from mathematicians with PhDs to solve. But in new tests, the most advanced AI models on the market got fewer than 2% of these problems right.<\/p>\n

<\/p>\n

Over the past decade, several AI tests have been developed to determine whether the answers returned by these models are actually correct. In many cases, AI models now easily pass these tests.<\/p>\n

For example, in the widely used Multitasking Language Understanding (MMLU) test, modern AI models correctly solve 98% of math problems.<\/p>\n

Most of these tests are aimed at testing the AI’s ability to solve math problems at the high school and college levels, Elliot Glaser, a mathematician at Epoch AI, and his colleagues wrote in a new paper posted to the arXiv preprint database.<\/p>\n\n

\"\"
\nThe researchers tested six modern AI models against the new benchmark, and the best result recorded by a single system was 2%. hh5800<\/em><\/p>\n

The new test set, called FrontierMath, targets higher-level reasoning. Epoch AI developed the questions with the help of math professors, including some winners of the Fields Medal, arguably the most prestigious award in mathematics. The problems cover a wide range of subfields, from number theory to algebraic geometry, and are available on the Epoch AI website.<\/p>\n

\u201cThese are extremely difficult problems,\u201d Terence Tao, a 2006 Fields Medalist and mathematician at the University of California, Los Angeles, wrote in a review of the problems for Epoch AI. \u201cI think that in the near term, essentially the only way to solve them without a true expert in the field is to combine a semi-expert, such as a PhD student in a related field, perhaps paired with some combination of modern AI and a variety of other algebraic packages.\u201d<\/p>\n\n

The problems were also unique, a step taken to ensure that none of the problems were already in the AI \u200b\u200bmodels\u2019 training data. When complex reasoning problems are included in the training data, it may appear that the AI \u200b\u200bis solving the problems, but in fact it already has a \u201ccheat sheet\u201d because it has been trained on the answers.<\/p>\n

The researchers tested six state-of-the-art AI models: Google’s Gemini 1.5 Pro (002), Anthropic’s Claude 3.5 Sonnet, and GPT4o’s o1-preview, o1-mini, and Grok-2 Beta. Gemini and Claude managed to solve 2% of the problems, which was only slightly better than o1-preview, o1-mini, and GPT-4o’s 1%. Grok-2 Beta failed to solve any of the problems correctly.<\/p>\n

However, the researchers cautioned that these rankings are misleading, as the low success rate means that one correct answer can have a huge impact on each model’s overall score.<\/p>\n

\u201cEven when the model got the answer right, that doesn\u2019t mean its reasoning was correct,\u201d the authors write. \u201cFor example, in one of these problems, running a few simple simulations was enough to make accurate guesses without any deeper mathematical understanding. However, the low overall accuracy of the models shows that such guessing strategies do not work in the vast majority of FrontierMath problems.\u201d<\/p>\n

The results show that the AI \u200b\u200bmodels currently lack research-grade mathematical reasoning, Epoch AI concluded. However, as the AI \u200b\u200bmodels evolve, these benchmark tests will provide a way to see if their reasoning abilities are improving.<\/p>\n

\u201cBy regularly evaluating state-of-the-art models and collaborating with the AI \u200b\u200bresearch community,\u201d the team said in a statement, \u201cwe aim to deepen our understanding of the capabilities and limitations of AI.\u201d<\/p>\n\n","protected":false},"excerpt":{"rendered":"

Current AI models struggle to solve research-grade math problems: The most advanced AI systems we have today solve just 2% of the hundreds of problems we face. According to the Epoch AI research institute, these problems typically take hours or days of work from mathematicians with PhDs to solve. But in new tests, the most advanced AI models on the market got fewer than 2% of these problems right.<\/p>\n","protected":false},"author":1,"featured_media":1324,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[238,40],"tags":[457,239],"class_list":{"0":"post-1323","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-exact-sciences","8":"category-philosophy","9":"tag-artificial-intelligence","10":"tag-mathematics"},"_links":{"self":[{"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/posts\/1323","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/comments?post=1323"}],"version-history":[{"count":3,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/posts\/1323\/revisions"}],"predecessor-version":[{"id":1388,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/posts\/1323\/revisions\/1388"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/media\/1324"}],"wp:attachment":[{"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/media?parent=1323"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/categories?post=1323"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/planeta21vek.ru\/artemiada\/eng\/wp-json\/wp\/v2\/tags?post=1323"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}