mongolol mongolol - 1 year ago 98
Scala Question

Removing Blank Strings from a Spark Dataframe

Attempting to remove rows in which a Spark dataframe column contains blank strings. Originally did

val df2 =
but it turns out many of these values are being encoded as

I'm stuck using Spark 1.3.1 and also cannot rely on DSL. (Importing spark.implicit_ isn't working.)

Answer Source

Removing things from a dataframe requires filter().

newDF = oldDF.filter("colName != ''")

or am I misunderstanding your question?

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download