teddy teddy - 3 years ago 80
Scala Question

How to bind variable in complex struct dataframes in spark

I have a code snippit like this:

val filteredDF = df.filter($"abc.color".equalTo("yellow") && $"abc.height".equalTo("5")).toDF()


The complex struct looks like:

{
abc:{
color: yellow,
height:5
}
}


What I want is to create a variable to represent abc, say

val table_name = "abc"


and create variables for color and height as well, say

val colorField = "color"
val heightField = "height"


Then how can I modify the code above to use those new variables?

I tried some ways like

df.filter(${table_name+"."+colorField}.equalTo("yellow")


and

df.filter(($""+table_name+"."+colorField).equalTo("yellow")


But neither works.

Answer Source

You can use Scala's String Interpolation

val table_name = "abc"
val colorField = "color"
val heightField = "height"

df.show
+----------+
|       abc|
+----------+
|[yellow,5]|
+----------+

import org.apache.spark.sql.functions.col

df.filter(col(s"${table_name}.${colorField}").equalTo("yellow")).show

+----------+
|       abc|
+----------+
|[yellow,5]|
+----------+

df.filter(col(s"${table_name}.${colorField}").equalTo("green")).show
+---+
|abc|
+---+
+---+
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download