psarka psarka - 3 months ago 15
Scala Question

Importing implicit methods in scalatest

I'm struggling to understand why implicit imports do not work as I expect them in scalatest. The simplified failing example (using spark, but I can make it fail with my custom class also) is as follows:

class FailingSpec extends FlatSpec with Matchers with MySparkContext {

val testSqlctx = sqlctx
import sqlctx.implicits._

"sql context implicts" should "work" in {
val failingDf = Seq(ID(1)).toDS.toDF
}
}


The MySparkContext trait creates and destroys spark context in
beforeAll
and
afterAll
, and makes
sqlctx
available (already having to reassign it to a local variable in order to import implicits is a puzzle, but maybe for a different time). The
.toDS
and
.toDF
are then implicit methods imported from
sqlctx.implicits
. Running the test results in a
java.lang.NullPointerException
.

If I move import into test block things work:

class WorkingSpec extends FlatSpec with Matchers with MySparkContext {

"sql context implicts" should "work" in {
val testSqlctx = sqlctx
import sqlctx.implicits._

val workingDf = Seq(ID(1)).toDS.toDF
}
}


Any ideas why can't I import implicits at the top level of the test class?

Answer

beforeAll runs before any of the tests, but does not run before the constructor for the class. The order of operations in the first snippet is:

  1. Constructor invoked, executing val testSqlctx = sqlctx and import sqlctx.implicits._

  2. beforeAll invoked

  3. Tests run

And the order of operations for the second snippet:

  1. beforeAll invoked

  2. Tests run, executing val testSqlctx = sqlctx and import sqlctx.implicits._

Assuming you give your SparkContext a default (null) value and initialize it in beforeAll, the first order of operations would try to use sqlctx when it is still null, thus causing the null pointer exception.