Learning from the failure of autonomous and intelligent systems: accidents, safety and sociotechnical sources of risk

Macrae, Carl

Abstract

Efforts to develop autonomous and intelligent systems (AIS) have exploded across a range of settings in recent years, from self-driving cars to medical diagnostic chatbots. These have the potential to bring enormous benefits to society but also have the potential to introduce new—or amplify existing—risks. As these emerging technologies become more widespread, one of the most critical risk management challenges is to ensure that failures of AIS can be rigorously analyzed and understood so that the safety of these systems can be effectively governed and improved. AIS are necessarily developed and deployed within complex human, social, and organizational systems, but to date there has been little systematic examination of the sociotechnical sources of risk and failure in AIS. Accordingly, this article develops a conceptual framework that characterizes key sociotechnical sources of risk in AIS by reanalyzing one of the most publicly reported failures to date: the 2018 fatal crash of Uber's self-driving car. Publicly available investigative reports were systematically analyzed using constant comparative analysis to identify key sources and patterns of sociotechnical risk. Five fundamental domains of sociotechnical risk were conceptualized—structural, organizational, technological, epistemic, and cultural—each indicated by particular patterns of sociotechnical failure. The resulting SOTEC framework of sociotechnical risk in AIS extends existing theories of risk in complex systems and highlights important practical and theoretical implications for managing risk and developing infrastructures of learning in AIS.

Citation

Macrae, C. (2021). Learning from the failure of autonomous and intelligent systems: accidents, safety and sociotechnical sources of risk. Risk Analysis, 42(9), 1999-2025. https://doi.org/10.1111/risa.13850

• Journal Article Type: Article
• Acceptance Date: Oct 13, 2021
• Online Publication Date: Nov 23, 2021
• Publication Date: Nov 23, 2021
• Deposit Date: Jan 4, 2022
• Publicly Available Date: Jan 5, 2022
• Journal: Risk Analysis
• Print ISSN: 0272-4332
• Electronic ISSN: 1539-6924
• Publisher: Wiley
• Peer Reviewed: Peer Reviewed
• Volume: 42
• Issue: 9
• Pages: 1999-2025
• DOI: https://doi.org/10.1111/risa.13850
• Keywords: Artificial intelligence; autonomous systems; organizational risk; system accident
• Public URL: https://nottingham-repository.worktribe.com/output/7164924
• Publisher URL: https://onlinelibrary.wiley.com/doi/10.1111/risa.13850