Fenil Suchak Fenil Suchak - 6 months ago 37
R Question

Model.Matrix returning infinite values

I have a data frame with no NA's(removed by na.omit()) with number of rows = 1,450,683 , while converting it into model.matrix to feed into glmnet the final matrix has some values that are Infinite.

str(train_again)
Classes 'tbl_df', 'tbl' and 'data.frame': 1450683 obs. of 24 variables:
$ vendor_id : Factor w/ 2 levels "1","2": 2 1 2 2 2 2 1 2 1 2 ...
$ passenger_count : int 1 1 1 1 1 6 4 1 1 1 ...
$ pickup_longitude : num -74 -74 -74 -74 -74 ...
$ pickup_latitude : num 40.8 40.7 40.8 40.7 40.8 ...
$ dropoff_longitude : num -74 -74 -74 -74 -74 ...
$ dropoff_latitude : num 40.8 40.7 40.7 40.7 40.8 ...
$ store_and_fwd_flag : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...
$ trip_duration : int 455 663 2124 429 435 443 341 1551 255 1274 ...
$ month : Factor w/ 6 levels "1","2","3","4",..: 3 6 1 4 3 1 6 5 5 5 ...
$ wday : Factor w/ 7 levels "Fri","Mon","Sat",..: 2 4 6 7 3 3 1 3 1 6 ...
$ hour : int 17 0 11 19 13 22 22 7 23 22 ...
$ work : Factor w/ 2 levels "FALSE","TRUE": 2 1 2 1 1 1 1 1 1 1 ...
$ jfk_trip : Factor w/ 2 levels "FALSE","TRUE": 1 1 1 1 1 1 1 1 1 1 ...
$ lg_trip : Factor w/ 2 levels "FALSE","TRUE": 1 1 1 1 1 1 1 1 1 1 ...
$ average.temperature: num 45.5 72.5 22 39 46.5 33.5 70.5 60 80 56.5 ...
$ rain : num 25 2 2 2 2 2 2 6 2 2 ...
$ s_fall : num 2 2 2 2 2 2 2 2 2 2 ...
$ s_depth : num 1 1 0.01 1 1 8 1 1 1 1 ...
$ total_distance : num 2009 2513 11061 1779 1615 ...
$ number_of_steps : int 5 6 16 4 5 5 5 17 2 6 ...
$ fastest_speed : num 43.9 27.3 51.9 27.2 41.5 ...
$ left_turns : int 1 2 5 2 2 1 1 4 0 2 ...
$ right_turns : int 1 2 7 1 2 3 3 9 1 2 ...
$ turns : int 1 2 9 1 3 3 2 6 0 3 ...


x = model.matrix(trip_duration~.,train_again) #here train_again is a data frame with no NA's
y = train_again$trip_duration
sum(is.infinite(x)) #gives output as 537


What could be the possible reason? Must there be something wrong with my original dataset?

Answer Source

Given your comments that one of your variables has an Inf (not NA) value, I would recommend something like:

(Using tidyverse since you already show you have a tibble.) Drop response variable:

predvars <- dplyr::select(train_again,-trip_duration)

Find all-finite rows (no NA, NaN, or Inf):

all_finite <- apply(is.finite(predvars),1,all)

You can use a one-sided formula in model.matrix():

x <- model.matrix(~.,predvars[all_finite,])
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download