Ciao AI: the Italian adaptation and validation of the Chatbot Usability Scale

Borsci, Simone; Prati, Elisa; Malizia, Alessio; Schmettow, Martin; Chamberlain, Alan; Federici, Stefano

Abstract

Chatbot-based tools are becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently, an eleven-item satisfaction inventory was developed (the ChatBot Usability Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and languages, i.e., English, German, Dutch, and Spanish. This scale forms a solid platform enabling designers to rapidly assess chatbots both during and after the design process. The present work aims to adapt and validate the BUS-11 inventory in Italian. A total of 1360 questionnaires were collected which related to a total of 10 Italian chatbot-based systems using the BUS-11 inventory and also using the lite version of the Usability Metrics for User eXperience for convergent validity purposes. The Italian version of the BUS-11 was adapted in terms of the wording of one item, and a Multi-Group Confirmatory Factorial Analysis was performed to establish the factorial structure of the scale and compare the effects of the wording adaptation. Results indicate that the adapted Italian version of the scale matches the expected factorial structure of the original scale. The Italian BUS-11 is highly reliable (Cronbach alpha: 0.921), and it correlates to other measures of satisfaction (e.g., UMUX-Lite, τb = 0.67; p <.001) by also offering specific insights regarding the chatbots’ characteristics. The Italian BUS-11 can be confidently used by chatbot designers to assess the satisfaction of their users during formative or summative tests.

• Journal Article Type: Article
• Acceptance Date: May 20, 2023
• Online Publication Date: Jun 1, 2023
• Publication Date: Jun 1, 2023
• Deposit Date: Jul 7, 2023
• Publicly Available Date: Jul 10, 2023
• Journal: Personal and Ubiquitous Computing
• Print ISSN: 1617-4909
• Electronic ISSN: 1617-4917
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 27
• Issue: 6
• Pages: 2161-2170
• DOI: https://doi.org/10.1007/s00779-023-01731-2
• Keywords: AI; Chatbots; Usability; Design; Language; Conversation; UX; Management Science and Operations Research; Computer Science Applications; Hardware and Architecture; Library and Information Sciences
• Public URL: https://nottingham-repository.worktribe.com/output/21378093
• Publisher URL: https://link.springer.com/article/10.1007/s00779-023-01731-2