Mission Reproducibility: An Investigation on Reproducibility Issues in Machine Learning and Information Retrieval Research

Abstract

This paper analyzes the most common problems limiting reproducibility of Information Retrieval research and provides researchers with insights and guidelines to improve the reproducibility of experiments and to allow the verification of obtained results. We conducted a study on 45 reproduction reports off 17 different papers, which have been published at renowned IR conferences. We analyzed the reports qualitatively and quantitatively and looked into the different insights from different groups. Occurring problems are classified into three problem families and 13 categories and afre then analyzed with respect to their influence on the reproduction process as well as on their frequency of appearance over time and per conference. Of these 17 different papers, 14 papers were reproducible to a certain degree without significant differences to the original results, but in many cases not the whole experiment was reproducible due to missing code, information or data. Also, we look at assumptions that were made when reproducing the different papers, as some experiment workflows were incomplete and information was missing. In addition, we propose recommendations to make machine learning research more reproducible and FAIR.