Dynamic updating of clinical survival prediction models in a changing environment

Tanner, Kamaryn T.; Keogh, Ruth H.; Coupland, Carol A. C.; Hippisley-Cox, Julia; Diaz-Ordaz, Karla

Abstract

Background
Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.

Methods
We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.

Results
In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.

Conclusions
We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

• Journal Article Type: Article
• Acceptance Date: Oct 17, 2023
• Online Publication Date: Dec 12, 2023
• Publication Date: Dec 12, 2023
• Deposit Date: Dec 19, 2023
• Publicly Available Date: Dec 19, 2023
• Journal: Diagnostic and Prognostic Research
• Print ISSN: 2397-7523
• Electronic ISSN: 2397-7523
• Publisher: BioMed Central
• Peer Reviewed: Peer Reviewed
• Volume: 7
• Article Number: 24
• DOI: https://doi.org/10.1186/s41512-023-00163-z
• Keywords: Model updating, Survival analysis, Dynamic model, Clinical prediction models
• Public URL: https://nottingham-repository.worktribe.com/output/28702127
• Publisher URL: https://link.springer.com/article/10.1186/s41512-023-00163-z
• Additional Information: Received: 16 June 2023; Accepted: 17 October 2023; First Online: 12 December 2023; : ; : This study protocol was reviewed and approved by the London School of Hygiene and Tropical Medicine Observational/Interventions Research Ethics Committee (Reference 26936/RR/27732). The study also received a favourable scientific opinion from the QResearch Scientific Committee. QResearch is a Research Ethics Approved Research Database, confirmed by the East Midlands - Derby Research Ethics Committee (REC reference 18/EM/0400).; : Not applicable.; : JHC is an unpaid director of QResearch, a not-for-profit organisation which is a partnership between the University of Oxford and EMIS Health who supply the QResearch database used for this work.