Liangpi Liangpi - 1 year ago 74
Scala Question

spark dataframe: how to explode a IntegerType column

val schema = StructType(Array(StructField("id", IntegerType, false),StructField("num", IntegerType, false)))

I want to generate continuous number from 0 to num by every id。
I don't know how to do ..

data and result here !!!

Answer Source

You can use UDF and explode function:

import org.apache.spark.sql.functions.{udf, explode}

val range = udf((i: Int) => (0 to i).toArray)
df.withColumn("num", explode(range($"num")))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download