Plot calibration stability across bootstrap replicates

A calibration (in)stability plot shows calibration curves for bootstrap models evaluated on original outcome. A stable model should produce boot calibration curves that differ minimally from the 'apparent' curve.

Usage

calibration_stability(
  x,
  calib_args,
  xlim,
  ylim,
  xlab,
  ylab,
  col,
  subset,
  plot = TRUE
)

Arguments

x

an object produced by validate with method = "boot_*" (or boot_optimism with method="boot")

calib_args

settings for calibration curve (see pmcalibration::pmcalibration). If unspecified settings are given by cal_defaults with 'eval' set to 100 (evaluate each curve at 100 points between min and max prediction).

xlim

x limits (default = c(0,1))

ylim

y limits (default = c(0,1))

xlab

a title for the x axis

ylab

a title for the y axis

col

color of lines for bootstrap models (default = grDevices::grey(.5, .3))

subset

vector of observations to include (row indices). If dataset is large fitting B curves is demanding. This can be used to select a random subset of observations.

plot

if FALSE just returns curves (see value)

Value

plots calibration (in)stability. Invisibly returns a list containing data for each curve (p=x-axis, pc=y-axis). The first element of this list is the apparent curve (original model on original outcome).

References

Riley, R. D., & Collins, G. S. (2023). Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal, 65(8), 2200302. doi:10.1002/bimj.202200302

Examples

# \donttest{
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation

calibration_stability(m1_iv)

# }