AMATU@Simpletext2024: Are LLMs Any Good for Scientific Leaderboard Extraction? : Notebook for the SimpleText Lab at CLEF 2024

Abstract

In this paper, we present our approach to solve the SOTA challenge of the SimpleText shared task at CLEF 2024. The objective of the challenge is to extract all (Task, Dataset, Metric, Score) tuples from scientific papers which report model score on benchmark datasets. In this work, we propose a rule-based classification model to identify papers that reports score information. We then apply different methods to extract TDMS using: (1) a baseline model from the literature, and (2) two Large Language Models (LLMs), GPT-3.5 and Mistral. Results show that the baseline model outperforms the LLMs in most cases, especially in zero-shot settings, with improvements seen in few-shot settings. Manual investigation shows that extracting TDMS from paper text is challenging, particularly for "Dataset" and "Score" extraction.