Empirical Quantification of Predictive Uncertainty Due to Model Discrepancy by Training with an Ensemble of Experimental Designs: An Application to Ion Channel Kinetics

Shuttleworth, Joseph G.; Lei, Chon Lok; Whittaker, Dominic G.; Windley, Monique J.; Hill, Adam P.; Preston, Simon P.; Mirams, Gary R.

Abstract

When using mathematical models to make quantitative predictions for clinical or industrial use, it is important that predictions come with a reliable estimate of their accuracy (uncertainty quantification). Because models of complex biological systems are always large simplifications, model discrepancy arises—models fail to perfectly recapitulate the true data generating process. This presents a particular challenge for making accurate predictions, and especially for accurately quantifying uncertainty in these predictions. Experimentalists and modellers must choose which experimental procedures (protocols) are used to produce data used to train models. We propose to characterise uncertainty owing to model discrepancy with an ensemble of parameter sets, each of which results from training to data from a different protocol. The variability in predictions from this ensemble provides an empirical estimate of predictive uncertainty owing to model discrepancy, even for unseen protocols. We use the example of electrophysiology experiments that investigate the properties of hERG potassium channels. Here, ‘information-rich’ protocols allow mathematical models to be trained using numerous short experiments performed on the same cell. In this case, we simulate data with one model and fit it with a different (discrepant) one. For any individual experimental protocol, parameter estimates vary little under repeated samples from the assumed additive independent Gaussian noise model. Yet parameter sets arising from the same model applied to different experiments conflict—highlighting model discrepancy. Our methods will help select more suitable ion channel models for future studies, and will be widely applicable to a range of biological modelling problems.

Citation

Shuttleworth, J. G., Lei, C. L., Whittaker, D. G., Windley, M. J., Hill, A. P., Preston, S. P., & Mirams, G. R. (2024). Empirical Quantification of Predictive Uncertainty Due to Model Discrepancy by Training with an Ensemble of Experimental Designs: An Application to Ion Channel Kinetics. Bulletin of Mathematical Biology, 86(1), Article 2. https://doi.org/10.1007/s11538-023-01224-6

• Journal Article Type: Article
• Acceptance Date: Oct 9, 2023
• Online Publication Date: Nov 24, 2023
• Publication Date: 2024-01
• Deposit Date: Dec 11, 2023
• Publicly Available Date: Dec 13, 2023
• Journal: Bulletin of Mathematical Biology
• Print ISSN: 0092-8240
• Electronic ISSN: 1522-9602
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 86
• Issue: 1
• Article Number: 2
• DOI: https://doi.org/10.1007/s11538-023-01224-6
• Keywords: Discrepancy, Mathematical model, Uncertainty quantification, Misspecification, Ion channel, Experimental design
• Public URL: https://nottingham-repository.worktribe.com/output/27854307
• Publisher URL: https://link.springer.com/article/10.1007/s11538-023-01224-6
• Additional Information: Received: 31 January 2023; Accepted: 9 October 2023; First Online: 24 November 2023