Tuning retrieval system effectiveness via the retrievability bias

Abstract

The performance of an information retrieval system depends on the document collection it has to work on and can be influenced to a certain degree by tuning the systems parameter values. There are several different measures to evaluate the effectiveness of retrieval systems.<br />All of them have in common that they require human relevance judgments to determine if search results are satisfying the information need of the queries. However, their creation is both very expensive and time consuming. There have been three fundamentally different approaches to create performance rankings without the need of these relevance judgments. This thesis first introduces relevant background information and then takes a closer look at the approach that shows the most potential in determining the top performers. It employs an unsupervised measure called retrievability which expresses how accessible documents are for the system. Experiments are conducted to compare various effectiveness measures to the retrievability bias at different system parameter values. This is done with three different retrieval methods on four different collections. Results show that while there is usually always a general correlation of high effectiveness and low retrievability bias observable, it largely depends on the length and/or quality of the user queries issued. In some cases a minimal retrievability bias does not indicate a good parameter setting for maximizing the effectiveness values. There might be several reasons for this behavior which still require further research.