ashwinsakthi ashwinsakthi -4 years ago 131
Scala Question

Why import implicit SqlContext.implicits._ after initializing SQLContext in a scala spark application

Why import implicit SqlContext.implicits._ after initializing SQLContext in a scala spark application.

when import is put outside the object , there seems to be an issue.

I am from a java back ground , where am not understanding the usage of import statements within a def object.

val sqlContext = new SQLContext(sc)

import sqlContext.implicits._
import sqlContext._

Answer Source

The import doesn't work externally because the implicits object is defined inside the SQLContext class:

* :: Experimental ::
* (Scala-specific) Implicit methods available in Scala for converting
* common Scala objects into `DataFrame`s.
*
* {{{
*   val sqlContext = new SQLContext(sc)
*   import sqlContext.implicits._
* }}}
@Experimental
@InterfaceStability.Evolving
object implicits extends SQLImplicits with Serializable {
  protected override def _sqlContext: SQLContext = self
}

Thus, only once you have an instance of SQLContext can you import the internal object definition containing all the implicits.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download