arivero arivero - 1 year ago 44
Scala Question

Why is Scala's Symbol not accepted as a column reference?

Trying the examples of Spark SQL, they seem to work well except when expressions are needed:

scala> val teenagers = people.where('age >= 10).where('age <= 19).select('name)
<console>:23: error: value >= is not a member of Symbol
val teenagers = people.where('age >= 10).where('age <= 19).select('name)

scala> val teenagers ='name)
<console>:23: error: type mismatch;
found : Symbol
required: org.apache.spark.sql.catalyst.expressions.Expression
val teenagers ='name)

It seems that I need an import not documented.

If I bulk importing everything

import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.dsl._
import org.apache.spark.sql.catalyst.errors._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
import org.apache.spark.sql.catalyst.types._
import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.execution
import org.apache.spark.sql.hive._

EDIT: ... and

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._

it works.

Answer Source

There is an implicit conversion you are lacking. import org.apache.spark.sql.SQLContext._ should probably work fine.

EDIT: import sqlContext._ where sqlContext is an instance of org.apache.spark.sql.SQLContext to be correct