Skip to main content

Research Repository

Advanced Search

A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

Borsci, Simone; Schmettow, Martin; Malizia, Alessio; Chamberlain, Alan; van der Velde, Frank

A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation Thumbnail


Simone Borsci

Martin Schmettow

Alessio Malizia

Frank van der Velde


The Bot Usability Scale (BUS) is a standardised tool to assess and compare the satisfaction of users after interacting with chatbots to support the development of usable conversational systems. The English version of the 15-item BUS scale (BUS-15) was the result of an exploratory factorial analysis; a confirmatory factorial analysis tests the replicability of the initial model and further explores the properties of the scale aiming to optimise this tool seeking for the stability of the original model, the potential reduction of items, and testing multiple language versions of the scale. BUS-15 and the usability metrics for user experience (UMUX-LITE), used here for convergent validity purposes, were translated from English to Spanish, German, and Dutch. A total of 1292 questionnaires were completed in multiple languages; these were collected from 209 participants interacting with an overall pool of 26 chatbots. BUS-15 was acceptably reliable; however, a shorter and more reliable solution with 11 items (BUS-11) emerged from the data. The satisfaction ratings obtained with the translated version of BUS-11 were not significantly different from the original version in English, suggesting that the BUS-11 could be used in multiple languages. The results also suggested that the age of participants seems to affect the evaluation when using the scale, with older participants significantly rating the chatbots as less satisfactory, when compared to younger participants. In line with the expectations, based on reliability, BUS-11 positively correlates with UMUX-LITE scale. The new version of the scale (BUS-11) aims to facilitate the evaluation with chatbots, and its diffusion could help practitioners to compare the performances and benchmark chatbots during the product assessment stage. This tool could be a way to harmonise and enable comparability in the field of human and conversational agent interaction.


Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., & van der Velde, F. (2023). A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation. Personal and Ubiquitous Computing, 27, 317–330.

Journal Article Type Article
Acceptance Date Jul 12, 2022
Online Publication Date Aug 4, 2022
Publication Date 2023-04
Deposit Date Aug 5, 2022
Publicly Available Date Aug 9, 2022
Journal Personal and Ubiquitous Computing
Print ISSN 1617-4909
Electronic ISSN 1617-4917
Publisher Springer Verlag
Peer Reviewed Peer Reviewed
Volume 27
Pages 317–330
Keywords Chatbots · Conversational agents · Usability · Artifcial Intelligence · Interaction
Public URL
Publisher URL


Chatbots Usability AI CHI User Interface Design HCI (1.4 Mb)

Publisher Licence URL

Copyright Statement
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

You might also like

Downloadable Citations