Subhod Lagade Subhod Lagade - 29 days ago 7
Scala Question

Scala Dataframe null check for columns

val new_df = df.filter($"type_interne" !== "" || $"type_interne" !== "null")


Give me error value
||
is not member of string

When i use
===
works well for filter

val new_df = df.filter($"type_interne" === "" || $"type_interne" === "null")

Answer

The problem seems to be the operator precedence, try to use braces:

 val new_df = df.filter(($"type_interne" !== "") || ($"type_interne" !== null))

you can also write it like this:

val new_df = df.filter(($"type_interne" !== "") or $"type_interne".isNotNull)