Benchmark Creation for Narrative Knowledge Delta Extraction Tasks: Can LLMs Help?

Abstract

Keeping up with state-of-the-art in science is increasingly difficult for researchers due to the current pace of publishing. Inspired by previous work, we address this challenge by formulating the task of Narrative Knowledge Delta (NKΔ) Extraction which focuses on identifying differences between pairs of scientific articles that tackle the same research problem, presented in a narrative form. We create two manually annotated ground truth datasets and one automatically generated dataset of NKΔ sentences. Using these datasets, we design and evaluate a NKΔ extraction approach from pairs of papers using four LLMs: GPT-4o, GPT-4o-mini, Llama3.1-8b, and Llama3.1-70b. We then apply a scientific fact-checking model to evaluate the LLMs’ NKΔ output using manually annotated data as ground truth claims. The results show a general improved performance in few-shot settings when examples from the automatically generated data are incorporated. However, our manual analysis reveals challenges and limitations in creating annotated data for evaluating NKΔ extraction by LLMs (Data, prompts, and code are available at: https://github.com/Alaa-Ebshihy/nkd_llm_2024).