Optimizing the tradeoff between cost and effectiveness in neural ranking

Abstract

The tradeoff between cost and effectiveness has always been an integral part of Information Retrieval. Since the early beginnings of IR, researchers have been concerned with creating the highest quality search system while abiding by the available hardware, scalability, or simply cost constraints. In recent years, the query to document text-retrieval task underwent an all-encompassing paradigm shift towards learned neural network based approaches. These -- usually large -- networks provide a great effectiveness boost over traditional indexing methods. However, these quality improvements come at large hardware and time costs and challenges to their scalability. This thesis contributes fundamental insights and approaches to retrieval and re- ranking using large scale language models. We developed a family of retrieval specific ranking models called the Transformer-Kernel architecture. The core of this architecture uses lightweight Transformer layers to contextualize text and Kernel- pooling for transparent matching. We propose several variants, specific for different sub-tasks of text retrieval, that expand the Pareto frontier between cost and effectiveness in each area. Our models also offer novel interpretability components to highlight contextualized term scores. We study the interaction between common ranking models and common retrieval datasets and highlight problems and provide improved dataset augmentations based on them. Furthermore, we propose widely applicable knowledge distillation procedures between different ranking architectures. The target architectures include both re-ranking and dense retrieval methods, that can replace traditional inverted indices. Our training processes, which use a trained teacher model to train a smaller or more efficient student model, again improve on the Pareto frontier -- as we increase the effectiveness of the efficient architectures, without changing their cost aspects at all. We conclude this thesis with a summary of our education initiatives that complement our research activities, by offering remote and open-source lectures, as well as proposing future directions based on the work in this thesis. We bring novel neural advances to the largest possible audience, both by lowering costs with acceptable effectiveness levels and providing the materials to lower the learning curve for this paradigm shift.