Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing modelsCitation formats

Standard

Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models. / Martin, Glen; Mamas, Mamas; Peek, Niels; Buchan, Iain; Sperrin, Matthew.

In: BMC Medical Research Methodology, Vol. 17, No. 1, 06.01.2017, p. 1.

Research output: Contribution to journalArticlepeer-review

Harvard

APA

Vancouver

Author

Bibtex

@article{093ac99323b74b91b5dcac7fdb830d1f,
title = "Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models",
abstract = "BackgroundClinical prediction models (CPMs) are increasingly deployed to support healthcare decisions, but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.MethodsSimulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new {\textquoteleft}local{\textquoteright} population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.ResultsWhile redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.ConclusionThis study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.",
keywords = "clinical prediction models, model aggregation, validation, computer simulation, contextual heterogeneity",
author = "Glen Martin and Mamas Mamas and Niels Peek and Iain Buchan and Matthew Sperrin",
year = "2017",
month = jan,
day = "6",
doi = "10.1186/s12874-016-0277-1",
language = "English",
volume = "17",
pages = "1",
journal = "BMC Medical Research Methodology ",
issn = "1471-2288",
publisher = "Springer Nature",
number = "1",

}

RIS

TY - JOUR

T1 - Clinical Prediction in Defined Populations: a simulation study investigating when and how to aggregate existing models

AU - Martin, Glen

AU - Mamas, Mamas

AU - Peek, Niels

AU - Buchan, Iain

AU - Sperrin, Matthew

PY - 2017/1/6

Y1 - 2017/1/6

N2 - BackgroundClinical prediction models (CPMs) are increasingly deployed to support healthcare decisions, but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.MethodsSimulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.ResultsWhile redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.ConclusionThis study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

AB - BackgroundClinical prediction models (CPMs) are increasingly deployed to support healthcare decisions, but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.MethodsSimulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.ResultsWhile redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.ConclusionThis study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

KW - clinical prediction models

KW - model aggregation

KW - validation

KW - computer simulation

KW - contextual heterogeneity

U2 - 10.1186/s12874-016-0277-1

DO - 10.1186/s12874-016-0277-1

M3 - Article

C2 - 28056835

VL - 17

SP - 1

JO - BMC Medical Research Methodology

JF - BMC Medical Research Methodology

SN - 1471-2288

IS - 1

ER -