Evaluating the Robustness of Collaborative Agents

Knott, Paul; Carroll, Micah; Devlin, Sam; Ciosek, Kamil; Hofmann, Katja; Dragan, Anca; Shah, Rohin

Abstract

Artificial agents trained by deep reinforcement learning will likely encounter novel situations after deployment that were never seen during training. Our agent must be robust to handle such situations well. However, if we cannot rely on the average training or validation reward as a metric, then how can we effectively evaluate robustness? We take inspiration from the practice of unit testing in software engineering. Specifically, we suggest that when designing AI agents that collaborate with humans, designers should search for potential edge cases in possible partner behavior and possible states encountered, and write tests which check that the behavior of the agent in these edge cases is reasonable. We apply this methodology to build a suite of unit tests for the Overcooked-AI environment, and use this test suite to evaluate three proposals for improving robustness. We find that the test suite provides significant insight into the effects of these proposals that were generally not revealed by looking solely at the average validation reward. For our full paper, see arxiv.org/abs/2101.05507.

Citation

Knott, P., Carroll, M., Devlin, S., Ciosek, K., Hofmann, K., Dragan, A., & Shah, R. (2021). Evaluating the Robustness of Collaborative Agents. In AAMAS '21: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (1560-1562)

• Conference Name: 20th International Conference on Autonomous Agents and Multiagent Systems
• Start Date: May 3, 2021
• End Date: May 7, 2021
• Acceptance Date: Feb 17, 2020
• Publication Date: 2021-05
• Deposit Date: Feb 5, 2021
• Publicly Available Date: May 21, 2021
• Publisher: ACM
• Pages: 1560-1562
• Book Title: AAMAS '21: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems
• ISBN: 9781450383073
• Public URL: https://nottingham-repository.worktribe.com/output/5291583
• Publisher URL: https://dl.acm.org/doi/10.5555/3463952.3464159
• Related Public URLs: https://aamas2021.soton.ac.uk/