SC. SC. - 12 days ago 5
R Question

R tryCatch neuralnet, unexpected output

I'm building a neural net for the Kaggle, "Ghost, Ghoul, Goblin" challenge.

I am sampling my (training) data X times (for illustration purposes X = 6) splitting it into a training and test set. I then run my neural net on each of the X (6) data sets and record the accuracy. I'm doing this so I can then compare the accuracy between different nets (one layer 3,4,5 hidden, two layers 3+3, 4+3 etc)

My neural net part of the code is

set.seed(26)
mysamples <- sapply(1:iterations, function(j) {
temp <- sample(1:371, size = insamplesize, replace = F)
})

nn3results <- data.frame(matrix(0, ncol = iterations, nrow = 1))

myseed <- 0
for (nnloop in 1:iterations){
myseed = myseed + 1
set.seed(myseed)
nn_idx <- mysamples[,nnloop]
nn_rep_train <- mydata[nn_idx,]
nn_rep_test <- mydata[-nn_idx,]
nn_rep <- neuralnet(Ghost+Ghoul+Goblin ~ bone_N + rot_N + hair_N + soul_N,
data=nn_rep_train, hidden=c(4))
mypredict <- compute(nn_rep, nn_rep_test[,2:5])$net.result
idx <- apply(mypredict, c(1), maxidx)
nn_rep_test$Pred <- as.factor(c('Ghost', 'Ghoul', 'Goblin')[idx])
print(paste("sim",nnloop,"=",round(sum(diag(table(nn_rep_test$type, nn_rep_test$Pred)))
/outsamplesize,5),"%","myseed = ",myseed))
nn3results[1,nnloop] <- sum(diag(table(nn_rep_test$type, nn_rep_test$Pred)))
}


Which performs exactly as I would expect, until iteration 5 where the neural net doesn't converge as follows

[1] "sim 1 = 0.74194 % myseed = 1"
[1] "sim 2 = 0.73118 % myseed = 2"
[1] "sim 3 = 0.75269 % myseed = 3"
[1] "sim 4 = 0.74194 % myseed = 4"
Error in nrow[w] * ncol[w] : non-numeric argument to binary operator
In addition: Warning messages:
1: algorithm did not converge in 1 of 1 repetition(s) within the stepmax
2: In is.na(weights) :
is.na() applied to non-(list or vector) of type 'NULL'


So not to worry. I have modified my code and put it inside
tryCatch
as follows.

myseed <- 0
for (nnloop in 1:iterations){
myseed = myseed + 1
set.seed(myseed)
nn_idx <- mysamples[,nnloop]
nn_rep_train <- mydata[nn_idx,]
nn_rep_test <- mydata[-nn_idx,]
tryCatch({
nn_rep <- neuralnet(Ghost+Ghoul+Goblin ~ bone_N + rot_N + hair_N + soul_N,
data=nn_rep_train, hidden=c(4))
},
error = function(e){nn3results[1,nnloop] <- -1},
warning = function(w){nn3results[1,nnloop] <- -1},
finally={
mypredict <- compute(nn_rep, nn_rep_test[,2:5])$net.result
idx <- apply(mypredict, c(1), maxidx)
nn_rep_test$Pred <- as.factor(c('Ghost', 'Ghoul', 'Goblin')[idx])
print(paste("sim",nnloop,"=",round(sum(diag(
table(nn_rep_test$type, nn_rep_test$Pred)))
/outsamplesize,5),"%","myseed = ",myseed))
nn3results[1,nnloop] <- sum(diag(table(nn_rep_test$type, nn_rep_test$Pred)))
})
}
nn3results


Now the unexpected result. Since simulation 5 returns an error I would expect the corresponding
nn3results
to have a value of
-1
due to
error = function(e){nn3results[1,nnloop] <- -1}
but now the whole code seems to work, even for simulation 5, when previously it didn't.

[1] "sim 1 = 0.74194 % myseed = 1"
[1] "sim 2 = 0.73118 % myseed = 2"
[1] "sim 3 = 0.75269 % myseed = 3"
[1] "sim 4 = 0.74194 % myseed = 4"
[1] "sim 5 = 0.70968 % myseed = 5"
[1] "sim 6 = 0.75269 % myseed = 6"
> nn3results
X1 X2 X3 X4 X5 X6
1 69 68 70 69 66 70


Seeds are the same. Results 1-4 are the same. Why does simulation 5 now work and not return -1?

As an aside any (constructive) code advice is always appreciated.

Answer

The code in the finally statement is always executed when there is an error and when there is no error. Therefore, in this case, you are computing predictions using the model from the previous step (model 4). You should place the code that evaluates your model inside the tryCatch block.

The finally statement is often used to ensure things are cleaned up. For example close a file. This should happen with or without an error.