Are there some ready to use libraries or packages for python or R to reduce the number of levels for large categorical factors?
I want to achieve something similar to R: "Binning" categorical variables but encode into the most frequently top-k factors and "other".
Here is an example in
data.table a bit, but it should be easy without
# Load data.table require(data.table) # Some data set.seed(1) dt <- data.table(type = factor(sample(c("A","B","C"), 10e3, replace = T)), weight = rnorm(n = 10e3, mean = 70, sd = 20)) # Decide the minimum frequency a level needs... min.freq <- 3350 # Levels that don't meet minumum frequency (using data.table) fail.min.f <- dt[, .N, type][N < min.freq, type] # Call all these level "Other" levels(dt$type)[fail.min.f] <- "Other"