Skip to contents

Score predictions for binary events

Source: R/score_binary.R
score_binary.Rd

Calculate scores summarizing discrimination/calibration of predictions against observed binary events. If score_fun is not defined when calling validate this function is used.

Usage

score_binary(y, p, ...)

Arguments

y

vector containing a binary outcome

p

vector of predictions

...

additional arguments. This function only supports calib_args as an optional argument. calib_args should contain arguments for pmcalibration::pmcalibration. If a calibration plot (apparent vs bias corrected calibration curves via cal_plot) is desired the argument 'eval' should be provided. This should be the points at which to evaluate the calibration curve on each boot resample or crossvalidation fold. A good option would be calib_args = list(eval = seq(min(p), max(p), length.out=100)); where p are predictions from the original model evaluated on the original data. Dots can be used to supply additional arguments to user-defined functions.

Value

a named vector of scores (see Details)

Details

The following measures are returned in a named vector.

C

the c-statistic (aka area under the ROC curve). Probability that randomly selected observation with y = 1 with have higher p compared to randomly selected y = 0.

Brier

mean squared error - mean((y - p)^2)

Intercept

Intercept from a logistic calibration model: glm(y ~ 1 + qlogis(p), family="binomial")

Slope

Slope from a logistic calibration model: glm(y ~ 1 + qlogis(p), family="binomial")

Eavg

average absolute difference between p and calibration curve (aka integrated calibration index or ICI).

E50

median absolute difference between p and calibration curve

E90

90th percentile absolute difference between p and calibration curve

Emax

maximum absolute difference between p and calibration curve

ECI

average squared difference between p and calibration curve. Estimated calibration index (Van Hoorde et al. 2015)

cal_plot

if eval is specified (via calib_args), values for plotting apparent and bias-corrected calibration curves are returned (see cal_plot). By default these are omitted from the summary printed (see summary.internal_validate).

Logistic calibration and other calibration metrics from non-linear calibration curves assessing 'moderate-calibration' (Eavg, E50, E90, Emax, ECI; see references) are calculated via the pmcalibration package. The default settings can be modified by passing calib_args to validate call. calib_args should be a named list corresponding to arguments to pmcalibration::pmcalibration.

References

Austin PC, Steyerberg EW. (2019) The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 38, pp. 1–15. https://doi.org/10.1002/sim.8281

Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., Van Calster, B. (2015). A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics, 54, pp. 283-93

Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina M., Steyerberg E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, pp. 167-176

Examples

p <- runif(100)
y <- rbinom(length(p), 1, p)
score_binary(y = y, p = p)
#>          C      Brier  Intercept      Slope       Eavg        E50        E90 
#> 0.83313325 0.16627180 0.10230809 1.09364881 0.01892466 0.02033055 0.03572761 
#>       Emax        ECI 
#> 0.03680029 0.05172805