<div class="csl-bib-body">
<div class="csl-entry">Jang, M., Kwon, D. S., & Lukasiewicz, T. (2022). BECEL: Benchmark for Consistency Evaluation of Language Models. In N. Calzolari, C.-R. Huang, & H. Kim (Eds.), <i>Proceedings of the 29th International Conference on Computational Linguistics</i> (pp. 3680–3696). International Committee on Computational Linguistics. http://hdl.handle.net/20.500.12708/192675</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/192675
-
dc.description.abstract
Behavioural consistency is a critical condition for a language model (LM) to become trustworthy like humans. Despite its importance, however, there is little consensus on the definition of LM consistency, resulting in different definitions across many studies. In this paper, we first propose the idea of LM consistency based on behavioural consistency and establish a taxonomy that classifies previously studied consistencies into several sub-categories. Next, we create a new benchmark that allows us to evaluate a model on 19 test cases, distinguished by multiple types of consistency and diverse downstream tasks. Through extensive experiments on the new benchmark, we ascertain that none of the modern pre-trained language models (PLMs) performs well in every test case, while exhibiting high inconsistency in many cases. Our experimental results suggest that a unified benchmark that covers broad aspects (i.e., multiple consistency types and tasks) is essential for a more precise evaluation.
en
dc.language.iso
en
-
dc.subject
behavioural consistency
en
dc.subject
language models
en
dc.title
BECEL: Benchmark for Consistency Evaluation of Language Models
en
dc.type
Inproceedings
en
dc.type
Konferenzbeitrag
de
dc.contributor.affiliation
University of Oxford, United Kingdom of Great Britain and Northern Ireland (the)
-
dc.contributor.affiliation
Korea Telecom (South Korea), Korea (the Republic of)
-
dc.description.startpage
3680
-
dc.description.endpage
3696
-
dc.type.category
Full-Paper Contribution
-
tuw.booktitle
Proceedings of the 29th International Conference on Computational Linguistics
-
tuw.relation.publisher
International Committee on Computational Linguistics