Classification instability plot — classification

Classification instability plot shows the relationship between original model estimated risk and the classification instability index (CII). The CII is the proportion of bootstrap replicates where the predicted class (0 if p <= threshold; 1 if p > threshold) is different to that obtained from the original model. Those with risk predictions around the threshold will exhibit elevated CII but an unstable model will exhibit high CII across a range of risk predictions. See Riley and Collins (2023).

Usage

classification_stability(x, threshold, xlim, ylim, xlab, ylab, pch, cex, col)

Arguments

x: an object produced by validate with method = "boot_\*" (or boot_optimism with method="boot")
threshold: estimated risks above the threshold get a predicted 'class' of 1, otherwise 0.
xlim: x limits (default = range of estimated risks)
ylim: y limits (default = c(0, maximum CII))
xlab: a title for the x axis
ylab: a title for the y axis
pch: plotting character (default = 16)
cex: controls point size (default = 1)
col: color of points (default = grDevices::grey(.5, .5))

Value

plots classification (in)stability. Invisibly returns estimates of CII for each observation.

References

Riley RD, Collins GS. (2023). Stability of clinical prediction models developed using statistical or machine learning methods. Biom J. doi:10.1002/bimj.202200302. Epub ahead of print.

Examples

set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation

classification_stability(m1_iv, threshold=.2)