Skip to main content

Research Repository

Advanced Search

A comparative analysis of interleaving methods for aggregated search

Chuklin, Aleksandr; Zhou, Ke; Schuth, Anne; De Rijke, Maarten

Authors

Aleksandr Chuklin

KE ZHOU KE.ZHOU@NOTTINGHAM.AC.UK
Assistant Professor

Anne Schuth

Maarten De Rijke



Abstract

A result page of a modern search engine often goes beyond a simple list of “10 blue links.” Many specific user needs (e.g., News, Image, Video) are addressed by so-called aggregated or vertical search solutions: specially presented documents, often retrieved from specific sources, that stand out from the regular organic Web search results. When it comes to evaluating ranking systems, such complex result layouts raise their own challenges. This is especially true for so-called interleaving methods that have arisen as an important type of online evaluation: by mixing results from two different result pages, interleaving can easily break the desired Web layout in which vertical documents are grouped together, and hence hurt the user experience. We conduct an analysis of different interleaving methods as applied to aggregated search engine result pages. Apart from conventional interleaving methods, we propose two vertical-aware methods: one derived from the widely used Team-Draft Interleaving method by adjusting it in such a way that it respects vertical document groupings, and another based on the recently introduced Optimized Interleaving framework. We show that our proposed methods are better at preserving the user experience than existing interleaving methods while still performing well as a tool for comparing ranking systems. For evaluating our proposed vertical-aware interleaving methods, we use real-world click data as well as simulated clicks and simulated ranking systems.

Citation

Chuklin, A., Zhou, K., Schuth, A., & De Rijke, M. (2015). A comparative analysis of interleaving methods for aggregated search. ACM Transactions on Information Systems, 33(2), doi:10.1145/2668120

Journal Article Type Article
Acceptance Date Sep 16, 2014
Online Publication Date Feb 25, 2015
Publication Date Feb 25, 2015
Deposit Date Sep 18, 2017
Print ISSN 1046-8188
Publisher Association for Computing Machinery (ACM)
Peer Reviewed Peer Reviewed
Volume 33
Issue 2
Article Number 5
DOI https://doi.org/10.1145/2668120
Public URL http://doi.acm.org/10.1145/2668120
Publisher URL https://dl.acm.org/citation.cfm?doid=2737813.2668120