Alex P Alex P - 3 months ago 25
R Question

Adding metrics to default train() output from the caret package

I would like to add additional metrics other than RMSE and Rsquared to the output of my linear model that I creating with the caret package. From what I understand, the code below will output the repeated cross-validated RMSE and Rsquared:

library(caret)
lm_reg1 <- train(log1p(mpg) ~ log1p(hp) + log1p(disp),
data = mtcars,
trControl = trainControl(method = "repeatedcv",
number = 10,
repeats = 10),
method = 'lm')
lm_reg


Output:

Linear Regression

32 samples
10 predictors

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 30, 29, 28, 29, 29, 28, ...
Resampling results:

RMSE Rsquared
0.1134972 0.8808378


I know I can modify the output to a custom metric by modifying the summaryFunction in trainControl and referring to it's name in the metric parameter. Here's an example of one that I created that calculates the MAPE of a log-log model:

mape <- function(actual, predicted){
mean(abs((actual - predicted)/actual))
}
mapeexpSummary <- function (data,
lev = NULL,
model = NULL) {
out <- mape(expm1(data$obs), expm1(data$pred))
names(out) <- "MAPEEXP"
out
}
lm_reg2 <- train(log1p(mpg) ~ log1p(hp) + log1p(disp),
data = mtcars,
trControl = trainControl(method = "repeatedcv",
number = 10,
summaryFunction = mapeexpSummary,
repeats = 10),
metric = 'MAPEEXP',
method = 'lm')
lm_reg2


Output:

Linear Regression

32 samples
10 predictors

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)
Summary of sample sizes: 28, 29, 29, 28, 28, 30, ...
Resampling results:

MAPEEXP
0.1022028


Is there any way to add them to a single output? I'm looking to save all of these values, but want to avoid creating two identical models to do so.

Answer

add in RMSE and Rsquared in your mapeexpSummary?

mapeexpSummary <- function (data,
    lev = NULL,
    model = NULL) {
    c(MAPEEXP=mape(expm1(data$obs), expm1(data$pred)),
        RMSE=sqrt(mean((data$obs-data$pred)^2)),
        Rsquared=summary(lm(pred ~ obs, data))$r.squared)
}
Comments