In a DataFrame object in Apache Spark (I'm using the Scala interface), if I'm iterating over its Row objects, is there any way to extract structure values by name?
I am using the below code to extract by name but I am facing problem on how to read the struct value .
If values had been of type string then we could have done this:
v1: struct (nullable = true)
| |-- level1: string (nullable = true)
| |-- level2: string (nullable = true)
| |-- level3: string (nullable = true)
| |-- level4: string (nullable = true)
| |-- level5: string (nullable = true)
You can access the
struct elements my first extracting another
Row (structs are modeled as another
Row in spark) from the toplevel
Row like this:
val level1 = row.getAs[Row]("struct").getAs[String]("level1")