Trust in Robot Benchmarking and Benchmarking for Trustworthy Robots

Thoduka, Santosh; Nair, Deebul; Caleb-Solly, Praminda; Dragone, Mauro; Cavallo, Filippo; Hochgeschwender, Nico

Abstract

Trustworthy evaluation of robots is necessary for them to be deployed and accepted in society. Scientific benchmarking competitions provide a way to evaluate robots outside of lab conditions. We propose a progressive and iterative benchmarking process through competitions, which incorporates an objective dataset-based evaluation, evaluation on a remote robot, and field evaluations for individual robot functionalities and complete tasks, in a cyclical process similar to the machine learning lifecycle, with a view to achieving trustworthy evaluation. The inclusion of out-of-distribution data, failure scenarios and user studies as part of the benchmarking process addresses the necessity to evaluate robot systems on non-functional qualities such as fault tolerance, adaptability, social acceptance, in addition to their functional abilities to improve trustworthiness.

Citation

Thoduka, S., Nair, D., Caleb-Solly, P., Dragone, M., Cavallo, F., & Hochgeschwender, N. (2024). Trust in Robot Benchmarking and Benchmarking for Trustworthy Robots. In M. I. Aldinhas Ferreira (Ed.), Producing Artificial Intelligent Systems: The Roles of Benchmarking, Standardisation and Certification (31-51). Springer. https://doi.org/10.1007/978-3-031-55817-7_3

• Online Publication Date: Jun 5, 2024
• Publication Date: Jun 5, 2024
• Deposit Date: Jul 21, 2024
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 31-51
• Series Title: Studies in Computational Intelligence
• Series Number: 1150
• Book Title: Producing Artificial Intelligent Systems: The Roles of Benchmarking, Standardisation and Certification
• ISBN: 978-3-031-55816-0
• DOI: https://doi.org/10.1007/978-3-031-55817-7_3
• Public URL: https://nottingham-repository.worktribe.com/output/37319001
• Publisher URL: https://link.springer.com/chapter/10.1007/978-3-031-55817-7_3
• Additional Information: First Online: 5 June 2024