Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers

Coppock, Harry; Nicholson, George; Kiskin, Ivan; Koutra, Vasiliki; Baker, Kieran; Budd, Jobie; Payne, Richard; Karoune, Emma; Hurley, David; Titcomb, Alexander; Egglestone, Sabrina; Tendero Cañadas, Ana; Butler, Lorraine; Jersakova, Radka; Mellor, Jonathon; Patel, Selina; Thornley, Tracey; Diggle, Peter; Richardson, Sylvia; Packham, Josef; Schuller, Björn W.; Pigoli, Davide; Gilmour, Steven; Roberts, Stephen; Holmes, Chris

Abstract

Recent work has reported that respiratory audio-trained AI classifiers can accurately predict SARS-CoV-2 infection status. However, it has not yet been determined whether such model performance is driven by latent audio biomarkers with true causal links to SARS-CoV-2 infection or by confounding effects, such as recruitment bias, present in observational studies. Here we undertake a large-scale study of audio-based AI classifiers as part of the UK government’s pandemic response. We collect a dataset of audio recordings from 67,842 individuals, with linked metadata, of whom 23,514 had positive polymerase chain reaction tests for SARS-CoV-2. In an unadjusted analysis, similar to that in previous works, AI classifiers predict SARS-CoV-2 infection status with high accuracy (ROC–AUC = 0.846 [0.838–0.854]). However, after matching on measured confounders, such as self-reported symptoms, performance is much weaker (ROC–AUC = 0.619 [0.594–0.644]). Upon quantifying the utility of audio-based classifiers in practical settings, we find them to be outperformed by predictions on the basis of user-reported symptoms. We make best-practice recommendations for handling recruitment bias, and for assessing audio-based classifiers by their utility in relevant practical settings. Our work provides insights into the value of AI audio analysis and the importance of study design and treatment of confounders in AI-enabled diagnostics.

Citation

Coppock, H., Nicholson, G., Kiskin, I., Koutra, V., Baker, K., Budd, J., Payne, R., Karoune, E., Hurley, D., Titcomb, A., Egglestone, S., Tendero Cañadas, A., Butler, L., Jersakova, R., Mellor, J., Patel, S., Thornley, T., Diggle, P., Richardson, S., Packham, J., …Holmes, C. (2024). Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers. Nature Machine Intelligence, 6, 229-242. https://doi.org/10.1038/s42256-023-00773-8

• Journal Article Type: Article
• Acceptance Date: Nov 19, 2023
• Online Publication Date: Feb 7, 2024
• Publication Date: 2024-02
• Deposit Date: Jul 10, 2023
• Publicly Available Date: Aug 8, 2024
• Journal: Nature Machine Intelligence
• Print ISSN: 2522-5839
• Electronic ISSN: 2522-5839
• Publisher: Springer Nature
• Peer Reviewed: Peer Reviewed
• Volume: 6
• Pages: 229-242
• DOI: https://doi.org/10.1038/s42256-023-00773-8
• Keywords: Diagnostic markers; Machine learning; SARS-CoV-2; Statistics
• Public URL: https://nottingham-repository.worktribe.com/output/22977064
• Publisher URL: https://www.nature.com/articles/s42256-023-00773-8
• Additional Information: Received: 19 January 2023; Accepted: 19 November 2023; First Online: 7 February 2024; : The authors declare no competing interests.; : This study has been approved by The National Statistician’s Data Ethics Advisory Committee (reference NSDEC(21)01) and the Cambridge South NHS Research Ethics Committee (reference 21/EE/0036) and Nottingham NHS Research Ethics Committee (reference 21/EM/0067). All participants reviewed the provided participant information and gave their informed consent to take part in the study.