Calibration drift in prognostic models

Review and evaluation of prognostic models developed using the CPRD database to assess calibration drift and model updating strategies

Why the research is needed

Prognostic models are tools that can be used to predict how a patient’s health condition may progress. They use information such as a patient’s age and medical history to make a prediction that is personalised to them. This prediction can be used by clinicians to help them decide how to best manage their care.

However, prognostic models are often based on old data and this can mean that their predictions are not relevant to new patients. The accuracy of prognostic models needs to be checked using recent data and if they are found to no longer give good predictions, these models should be updated.

What is already known about the subject

The accuracy of prognostic models deteriorating over time (calibration drift) is a common problem. This can be due to new treatments and healthcare policies leading to improved patient outcomes. It is therefore recommended that the performance of prognostic models is monitored to ensure that they continue to give accurate predictions and reflect current clinical practice.

Who we will be working with

We will be working with PPI groups and researchers from the University of Manchester.

How patients and the public are involved

We will be working with a PPI group to discuss the findings of our study and plan how to best share the results of this research with others. We will also be working with the University of Leicester PPI-SMART team (Patient and Public Involvement with Statistical Methods and Research Techniques) who specialise in conducting PPI for statistical methodology projects.

What we will do

We will be reviewing three prognostic models for chronic obstructive pulmonary disease (COPD) that predict a patient’s risk of death up to 10 years after their diagnosis of COPD. These models were developed using the Clinical Research Practice Datalink (CPRD) several years ago.

We will be using more recent data from the CPRD to test how accurate these models are at making predictions for current COPD patients. If we find any of these models no longer give accurate predictions, we will update them accordingly to ensure that they are relevant to current clinical practice.

What the benefits will be

It is essential that prognostic models give accurate predictions in order for them to be useful clinically. This study will demonstrate how model performance can be assessed in new data and will compare a range of statistical methods that can be used for model updating. This will lead to more accurate risk predictions for COPD mortality.

When the findings will be available

We will be sharing our findings in a journal article which we expect to be published by the end of 2024.

How we are planning for implementation

We will share our findings of the updated prognostic models for the three COPD case studies to ensure more accurate risk predictions are made for patients with this condition. We will also use these exemplars to further promote the statistical methodology required to update prognostic models to persuade others to use this approach in other disease areas.

Contact

Sarah Booth, Lecturer in Medical Statistics, University of Leicester, sarah.booth@le.ac.uk