Liangpi Liangpi - 4 months ago 23
Scala Question

spark dataframe: how to explode a IntegerType column

val schema = StructType(Array(StructField("id", IntegerType, false),StructField("num", IntegerType, false)))


I want to generate continuous number from 0 to num by every id。
I don't know how to do ..
Thanks

data and result here !!!

Answer Source

You can use UDF and explode function:

import org.apache.spark.sql.functions.{udf, explode}

val range = udf((i: Int) => (0 to i).toArray)
df.withColumn("num", explode(range($"num")))