Feynman27 Feynman27 - 1 year ago 84
Scala Question

Replace all ":" with "_" in Spark dataframe

I'm trying to replace all instances of ":" --> "_" in a single column of a Spark dataframe. I'm trying to do this with:

val url_cleaner = (s:String) => {
val url_cleaner_udf = udf(url_cleaner)
val df = old_df.withColumn("newCol", url_cleaner_udf(old_df("oldCol")) )

But I keep getting the error:

SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 4 times, most recent failure: Lost task 0.3 in stage 25.0 (TID 692, ip-10-81-194-29.ec2.internal): java.lang.NullPointerException

Where am I going wrong in the udf?

Answer Source

Probably you've got some nulls in this column.


val urlCleaner = (s:String) => {
   if (s == null) null else s.replaceAll(":","_")

You can also use regexp_replace(col("newCol"), ":", "_") instead of own function

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download