Gevorg Gevorg - 7 months ago 51
Scala Question

Filter spark/scala dataframe if column is present in set

I'm using Spark 1.4.0, this is what I have so far:

data.filter($"myColumn".in(lit("A"), lit("B"), lit("C"), ...))

The function lit converts a literal to a column.

Ideally I would put my A, B, C in a Set and check like this:

val validValues = Set("A", "B", "C", ...)

What's the correct syntax? Are there any alternative concise solutions?


Spark 1.4 or older:

val validValues = Set("A", "B", "C").map(lit(_))
data.filter($"myColumn".in(validValues.toSeq: _*))

Spark 1.5 or newer:

val validValues = Set("A", "B", "C")
data.filter($"myColumn".isin(validValues.toSeq: _*))