<div class="csl-bib-body">
<div class="csl-entry">Althammer, S., Hofstätter, S., Verberne, S., & Hanbury, A. (2022). TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval. In <i>CIKM ’22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management</i> (pp. 3801–3805). https://doi.org/10.1145/3511808.3557714</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/137006
-
dc.description.abstract
Robust test collections are crucial for Information Retrieval research. Recently there is a growing interest in evaluating retrieval systems for domain-specific retrieval tasks, however these tasks often lack a reliable test collection with human-annotated relevance assessments following the Cranfield paradigm. In the medical domain, the TripClick collection was recently proposed, which contains click log data from the Trip search engine and includes two click-based test sets. However the clicks are biased to the retrieval model used, which remains unknown, and a previous study shows that the test sets have a low judgement coverage for the Top-10 results of lexical and neural retrieval models. In this paper we present the novel, relevance judgement test collection TripJudge for TripClick health retrieval. We collect relevance judgements in an annotation campaign and ensure the quality and reusability of TripJudge by a variety of ranking methods for pool creation, by multiple judgements per query-document pair and by an at least moderate inter-annotator agreement. We compare system evaluation with TripJudge and TripClick and find that that click and judgement-based evaluation can lead to substantially different system rankings.
en
dc.language.iso
en
-
dc.subject
health retrieval
en
dc.subject
relevance judgements
en
dc.subject
test collections
en
dc.title
TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval
en
dc.type
Inproceedings
en
dc.type
Konferenzbeitrag
de
dc.relation.isbn
9781450392365
-
dc.description.startpage
3801
-
dc.description.endpage
3805
-
dc.type.category
Poster Contribution
-
tuw.booktitle
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management
-
tuw.researchTopic.id
I4a
-
tuw.researchTopic.name
Information Systems Engineering
-
tuw.researchTopic.value
100
-
tuw.publication.orgunit
E194-04 - Forschungsbereich Data Science
-
tuw.publisher.doi
10.1145/3511808.3557714
-
dc.description.numberOfPages
5
-
tuw.author.orcid
0000-0002-7149-5843
-
tuw.event.name
31st ACM International Conference on Information and Knowledge Management
en
tuw.event.startdate
17-10-2022
-
tuw.event.enddate
21-10-2022
-
tuw.event.online
Hybrid
-
tuw.event.type
Event for scientific audience
-
tuw.event.country
US
-
tuw.event.presenter
Hofstätter, Sebastian
-
wb.sciencebranch
Informatik
-
wb.sciencebranch.oefos
1020
-
wb.sciencebranch.value
100
-
item.openairetype
Inproceedings
-
item.openairetype
Konferenzbeitrag
-
item.grantfulltext
none
-
item.cerifentitytype
Publications
-
item.cerifentitytype
Publications
-
item.languageiso639-1
en
-
item.openairecristype
http://purl.org/coar/resource_type/c_18cf
-
item.openairecristype
http://purl.org/coar/resource_type/c_18cf
-
item.fulltext
no Fulltext
-
crisitem.author.dept
E194-04 - Forschungsbereich E-Commerce
-
crisitem.author.dept
E194-04 - Forschungsbereich E-Commerce
-
crisitem.author.dept
E194-04 - Forschungsbereich E-Commerce
-
crisitem.author.orcid
0000-0002-7149-5843
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering
-
crisitem.author.parentorg
E194 - Institut für Information Systems Engineering