The IMO Small Challenge: Not-Too-Hard Olympiad Math Datasets for LLMs

Abstract

We introduce the IMO Small Challenge (IMOSC), as opposed to the IMO Grand Challenge: A text-only, natural-language dataset consisting of mathematical problems from various mathematical competitions. The IMOSC dataset exceeds the difficulty level of current datasets that are widely used for LLM evaluation, such as the MATH dataset, while not being too challenging for the current generation of LLMs. The IMOSC currently contains a carefully curated collection of the easiest possible problems from difficult competitions, such as the International Mathematical Olympiad (IMO). Problem hardness is measured by applying a mixture of (objective and subjective) difficulty filters to the original problems. We release the full dataset under the link below to encourage transparent evaluation of LLMs and theorem provers toward their mathematical proof-generating abilities: www.imo-small-challenge.io