Swetha Swetha - 1 year ago 289
SQL Question

How to check isEmpty on Column Data Spark scala

My data looks like :


How do I check if the col3 is empty on query in spark sql ? I tried to explode but when I do that the empty array rows are disappearing. Can some suggest me a way to do this.

I tried :

val homeSet = result.withColumn("subscriptionProvider", explode($"subscriptionProvider"))

where subscriptionProvider(WrappedArray()) is the column having array of values but some arrays can be empty. I need to get the subscriptionProvider with null values and subscriptionProvider array has "Comcast"

Answer Source


import org.apache.spark.sql.functions._

val tmp = df.withColumn("subscriptionProvider", 
  when(size($"subscriptionProvider") !== 0, $"subscriptionProvider").otherwise(array(lit(null).cast("string"))))

tmp.withColumn("subscriptionProvider", explode($"subscriptionProvider"))