Skip to contents

Calculate bias-corrected scores via cross-validation

Source: R/crossval.R
crossval.Rd

Estimate bias-corrected scores via cross-validation. CV is used to calculate optimism which is then subtracted from apparent scores and to calculate average performance in the out of sample (held out) data. This function is called by validate.

Usage

crossval(data, outcome, model_fun, pred_fun, score_fun, k = 10, ...)

Arguments

data

the data used in developing the model. Should contain all variables considered (i.e., even those excluded by variable selection in the development sample)

outcome

character denoting the column name of the outcome in data.

model_fun

a function that takes at least one argument, data. This function should implement the entire model development procedure (i.e., hyperparameter tuning, variable selection, imputation). Additional arguments can be provided via .... This function should return an object that works with pred_fun.

pred_fun

function that takes at least two arguments, model and data. This function should return a numeric vector of predicted probabilities of the outcome with the same length as the number of rows in data so it is important to take into account how missing data is treated (e.g., predict.glm omits predictions for rows with missing values).

score_fun

a function to calculate the metrics of interest. If this is not specified score_binary is used.

k

number of folds. Typically scores need >> 2 observations to be calculated so folds should be chosen with this in mind.

...

additional arguments for model_fun, pred_fun, and/or score_fun.

Value

a list of class internal_cv containing:

  • apparent - scores calculated on the original data using the original model.

  • optimism - estimates of optimism for each score (average difference in score for training data vs test data on each fold) which can be subtracted from 'apparent' performance calculated using the original model on the original data.

  • cv_optimism_corrected - 'bias corrected' scores (apparent - optimism). This is what is produced by rms::validate, rms::predab.resample.

  • cv_average - average of scores calculated on the test (held out) data. This is the metric described in Steyerberg et al. (2001).

  • indices - indices used to define test set on each fold.

References

Steyerberg, E. W., Harrell Jr, F. E., Borsboom, G. J., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001). Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of clinical epidemiology, 54(8), 774-781.

Examples

library(pminternal)
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 1000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.186
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
#m1 <- glm(y ~ x1 + x2, data=dat, family="binomial")

model_fun <- function(data, ...){
  glm(y ~ x1 + x2, data=data, family="binomial")
}

pred_fun <- function(model, data, ...){
  predict(model, newdata=data, type="response")
}

# CV Corrected = Apparent - CV Optimism
# CV Average = average score in held out fold
crossval(data=dat, outcome="y", model_fun=model_fun, pred_fun=pred_fun, k=10)
#>                    C   Brier Intercept  Slope   Eavg    E50    E90  Emax    ECI
#> Apparent      0.7964  0.1262   6.6e-15  1.000  0.020  0.016  0.033  0.10  0.062
#> CV Optimism  -0.0024 -0.0013   2.7e-02 -0.033 -0.037 -0.029 -0.077 -0.11 -0.561
#> CV Corrected  0.7988  0.1275  -2.7e-02  1.033  0.056  0.045  0.110  0.21  0.622
#> CV Average    0.7989  0.1274  -2.7e-02  1.033  0.056  0.045  0.110  0.21  0.624