Plot prediction stability across bootstrap replicates
Source:R/stability_plot.R
prediction_stability.Rd
A prediction (in)stability plot shows estimated risk probabilities from models developed on resampled data evaluated on the original development data as a function of the 'apparent' prediction (prediction from original/development model evaluated on original data). A stable model should produce points that exhibit minimal dispersion. See Riley and Collins (2023).
Usage
prediction_stability(
x,
bounds = 0.95,
smooth_bounds = FALSE,
xlab,
ylab,
pch,
cex,
col,
lty,
span
)
Arguments
- x
an object produced by
validate
with method = "boot_\*" (orboot_optimism
with method="boot")- bounds
width of the 'stability interval' (percentiles of the bootstrap model predictions). NULL = do not add bounds to plot.
- smooth_bounds
if TRUE, use
loess
to smooth the bounds (default = FALSE)- xlab
a title for the x axis
- ylab
a title for the y axis
- pch
plotting character (default = 16)
- cex
controls point size (default = 0.4)
- col
color of points (default = grDevices::grey(.5, .5))
- lty
line type for bounds (default = 2)
- span
controls the degree of smoothing (see
loess
; default = 0.75)
Value
plots prediction (in)stability. The stability bounds are not smoothed. Invisibly returns stability matrix (where column 1 are original predictions) that can be used for creating plots with other packages/software.
References
Riley RD, Collins GS. (2023). Stability of clinical prediction models developed using statistical or machine learning methods. Biom J. doi:10.1002/bimj.202200302. Epub ahead of print.
Examples
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor
# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")
# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation
prediction_stability(m1_iv)