Presentation - Poster Presentation

English

Staudinger, Moritz 
Kusa, Wojciech 
Piroi, Florina 
Lipani, Aldo 
Hanbury, Allan 

E194-04 - Forschungsbereich Data Science 
E058-06 - Fachbereich Zentrum für Forschungsdatenmanagement 

9-Nov-2024

ML in PL Conference 2024 

7-Nov-2024 - 10-Nov-2024

Warsaw, Poland

Systematic Reviews; Boolean query; LLMs; query generation

Systematic literature reviews (SLRs) are a cornerstone of academic research, yet they are often labour-intensive and time-consuming due to the detailed literature curation process. The advent of generative AI and large language models (LLMs) promises to revolutionize this process by assisting researchers in several tedious tasks, one of them being the generation of effective Boolean queries that will select the publications to consider including in a review. This paper presents a extensive study of Boolean query generation using LLMs for systematic reviews, reproducing and extending the work of Wang et al. and Alaniz et al. Our study investigates the replicability and reliability of results achieved using ChatGPT and compares its performance with open-source alternatives like Mistral and Zephyr to provide a comprehensive analysis of LLMs for query generation. Therefore, we implemented a pipeline, which automatically creates a Boolean query for a given review topic by using a previously selected LLM, retrieves all documents for this query from the PubMed database and then evaluates the results. With this pipeline we first assess whether the results obtained using ChatGPT for query generation are reproducible and consistent. We then generalize our results by analyzing and evaluating open-source models and evaluating their efficacy in generating Boolean queries.

Information Systems Engineering: 100%

1020 - Informatik: 90%
5020 - Wirtschaftswissenschaften: 10%