<div class="csl-bib-body">
<div class="csl-entry">Staudinger, M., Kusa, W., Piroi, F., Lipani, A., & Hanbury, A. (2024, November 9). <i>Beyond ChatGPT: A Reproducibility and Generalizability Study of Large Language Models for Query Generation</i> [Poster Presentation]. ML in PL Conference 2024, Warsaw, Poland.</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/208253
-
dc.description
Systematic literature reviews (SLRs) are a cornerstone of academic research, yet they are often labour-intensive and time-consuming due to the detailed literature curation process. The advent of generative AI and large language models (LLMs) promises to revolutionize this process by assisting researchers in several tedious tasks, one of them being the generation of effective Boolean queries that will select the publications to consider including in a review. This paper presents a extensive study of Boolean query generation using LLMs for systematic reviews, reproducing and extending the work of Wang et al. and Alaniz et al. Our study investigates the replicability and reliability of results achieved using ChatGPT and compares its performance with open-source alternatives like Mistral and Zephyr to provide a comprehensive analysis of LLMs for query generation. Therefore, we implemented a pipeline, which automatically creates a Boolean query for a given review topic by using a previously selected LLM, retrieves all documents for this query from the PubMed database and then evaluates the results. With this pipeline we first assess whether the results obtained using ChatGPT for query generation are reproducible and consistent. We then generalize our results by analyzing and evaluating open-source models and evaluating their efficacy in generating Boolean queries.
-
dc.language.iso
en
-
dc.subject
Systematic Reviews
en
dc.subject
Boolean query
en
dc.subject
LLMs
en
dc.subject
query generation
en
dc.title
Beyond ChatGPT: A Reproducibility and Generalizability Study of Large Language Models for Query Generation
en
dc.type
Presentation
en
dc.type
Vortrag
de
dc.type.category
Poster Presentation
-
tuw.researchTopic.id
I4
-
tuw.researchTopic.name
Information Systems Engineering
-
tuw.researchTopic.value
100
-
tuw.publication.orgunit
E194-04 - Forschungsbereich Data Science
-
tuw.publication.orgunit
E058-06 - Fachbereich Zentrum für Forschungsdatenmanagement
-
tuw.author.orcid
0000-0002-5164-2690
-
tuw.author.orcid
0000-0003-4420-4147
-
tuw.author.orcid
0000-0001-7584-6439
-
tuw.author.orcid
0000-0002-7149-5843
-
tuw.event.name
ML in PL Conference 2024
-
tuw.event.startdate
07-11-2024
-
tuw.event.enddate
10-11-2024
-
tuw.event.online
On Site
-
tuw.event.type
Event for scientific audience
-
tuw.event.place
Warsaw
-
tuw.event.country
PL
-
tuw.event.presenter
Staudinger, Moritz
-
wb.sciencebranch
Informatik
-
wb.sciencebranch
Wirtschaftswissenschaften
-
wb.sciencebranch.oefos
1020
-
wb.sciencebranch.oefos
5020
-
wb.sciencebranch.value
90
-
wb.sciencebranch.value
10
-
item.languageiso639-1
en
-
item.openairetype
conference poster not in proceedings
-
item.grantfulltext
none
-
item.fulltext
no Fulltext
-
item.cerifentitytype
Publications
-
item.openairecristype
http://purl.org/coar/resource_type/c_18co
-
crisitem.author.dept
E194-04 - Forschungsbereich Data Science
-
crisitem.author.dept
E194-04 - Forschungsbereich Data Science
-
crisitem.author.dept
E058-06 - Fachbereich Zentrum für Forschungsdatenmanagement
-
crisitem.author.dept
E194-04 - Forschungsbereich Data Science
-
crisitem.author.dept
E194-04 - Forschungsbereich Data Science
-
crisitem.author.orcid
0000-0002-5164-2690
-
crisitem.author.orcid
0000-0003-4420-4147
-
crisitem.author.orcid
0000-0001-7584-6439
-
crisitem.author.orcid
0000-0002-7149-5843
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
-
crisitem.author.parentorg
E058 - Forschungs-, Technologie- und Innovationssupport
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
