Cross-validation

Methods for measuring the performance of a predictive model on sets of test data in Bradley-Terry model from psychotree, Generalized Linear and Generalized Nonlinear models from gnm, and Plackett-Luce model from PlackettLuce. The package that provides the model (gnm, PlackettLuce or psychotree) must be attached with library() before calling crossvalidation().

crossvalidation(formula, data, k = 10, folds = NULL, seed = NULL, ...)

# S3 method for class 'bttree'
AIC(object, newdata = NULL, ...)

# S3 method for class 'bttree'
deviance(object, newdata = NULL, ...)

# S3 method for class 'pltree'
deviance(object, newdata = NULL, ...)

# S3 method for class 'gnm'
AIC(object, newdata = NULL, ...)

# S3 method for class 'gnm'
deviance(object, newdata = NULL, ...)

Arguments

formula

an object of class "formula" (or one that can be coerced to that class): a symbolic description of the model to be fitted, of the form \(y ~ x1 + ... + xn\)

data

a data frame (or object coercible by as.data.frame to a data frame) containing the variables in the model

k

an integer for the number of bins in the cross-validation

folds

an optional vector or list specifying the k-folds in the cross-validation. A vector gives, for each row of data, the fold (1 to k) in which that row is held out for testing. A list gives, for each fold, the row positions of data used for training, as returned by caret::createFolds(..., returnTrain = TRUE). Rows dropped for missing values are removed from the folds

seed

integer, the seed used to create the folds. If NULL (the default), folds are drawn from the current random number stream. When a seed is given, the caller's random number state is restored on exit

...

additional arguments passed to the methods of the chosen model

object

a model object

newdata

a data.frame with test data

Value

an object of class gosset_cv with the cross-validation goodness-of-fit estimates, which are:

AIC

Akaike Information Criterion

deviance

Model deviance

logLik

Log-Likelihood

MaxLik

Maximum likelihood pseudo R-squared

CraggUhler

Cragg and Uhler's pseudo R-squared

McFadden

McFadden pseudo R-squared

kendallTau

the Kendall correlation coefficient

References

Elder J. F. (2003). Journal of Computational and Graphical Statistics, 12(4), 853–864. doi:10.1198/1061860032733

James G., et al. (2013). doi:10.1007/978-1-4614-7138-7

Whitlock M. C. (2005). Journal of Evolutionary Biology, 18(5), 1368–1373. doi:10.1111/j.1420-9101.2005.00917.x

See also

bttree, gnm, pltree

Other model selection functions: btpermute()

Author

Kauê de Sousa, Jacob van Etten and David Brown

Examples

# \donttest{
# Generalized Linear Models
if (require("gnm")) {
data("airquality")

cv = crossvalidation(Temp ~ Wind + Solar.R,
                      data = airquality,
                      k = 3,
                      seed = 999,
                      family = poisson())
}                    
# Plackett-Luce Model
if(require("PlackettLuce")) {
# beans data from PlackettLuce
data("beans", package = "PlackettLuce")

G = rank_tricot(data = beans,
                 items = c(1:3),
                 input = c(4:5),
                 additional.rank = beans[c(6:8)],
                 group = TRUE)

beans = cbind(G, beans)

# take seasons as bins
k = length(unique(beans$season))
folds = as.integer(as.factor(beans$season))

cv = crossvalidation(G ~ maxTN,
                      data = beans,
                      k = k,
                      folds = folds,
                      minsize = 100)
}
# }