POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling

Liu, Zeyang; Zhou, Ke; Mao, Jiaxin; Wilson, Max L.

Abstract

Conversational search systems, such as Google Assistant and Microsoft Cortana, provide a new search paradigm where users are allowed, via natural language dialogues, to communicate with search systems. Evaluating such systems is very challenging since search results are presented in the format of natural language sentences. Given the unlimited number of possible responses, collecting relevance assessments for all the possible responses is infeasible. In this paper, we propose POSSCORE, a simple yet effective automatic evaluation method for conversational search. The proposed embedding-based metric takes the influence of part of speech (POS) of the terms in the response into account. To the best knowledge, our work is the first to systematically demonstrate the importance of incorporating syntactic information, such as POS labels, for conversational search evaluation. Experimental results demonstrate that our metrics can correlate with human preference, achieving significant improvements over state-of-the-art baseline metrics.

Citation

Liu, Z., Zhou, K., Mao, J., & Wilson, M. L. (2021, November). POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling. Presented at CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event Queensland Australia

• Presentation Conference Type: Edited Proceedings
• Conference Name: CIKM '21: The 30th ACM International Conference on Information and Knowledge Management
• Start Date: Nov 1, 2021
• End Date: Nov 5, 2021
• Acceptance Date: Aug 8, 2021
• Online Publication Date: Oct 30, 2021
• Publication Date: Oct 26, 2021
• Deposit Date: Mar 20, 2025
• Publicly Available Date: Apr 17, 2025
• Publisher: Association for Computing Machinery (ACM)
• Peer Reviewed: Peer Reviewed
• Pages: 1119-1129
• Book Title: CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management
• ISBN: 9781450384469
• DOI: https://doi.org/10.1145/3459637.3482463
• Public URL: https://nottingham-repository.worktribe.com/output/46737770
• Publisher URL: https://dl.acm.org/doi/10.1145/3459637.3482463