Georg Heiler Georg Heiler - 2 days ago 5
Scala Question

Spark testing base strange validation failure

When I execute

sbt test
in order to test a spark application utilizing
spark-testing-base


Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0)) did not equal Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))


for a test case of

val input: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0))
val expectedOutput: Map[String, Any] = Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))
val result = SimpleContext.runTheJOb(session, input)


Minimal example can be found https://github.com/geoHeil/apache-spark-restAPI-example

edit



the whole test case directly

class SimpleTest extends FunSuite with SharedSparkContext with DatasetSuiteBase {

test("SimpleContext should multiply input numbers by 3") {
val session = spark

val input: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0))
val expectedOutput: Map[String, Any] = Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))
val result = SimpleContext.runTheJOb(session, input)
assert(expectedOutput === result)
}
}

Answer

The problem you are having is that Scala Arrays, being Java arrays under the hood, are not comparable using == (see this answer for an explanation).

Example:

scala> Array(1,2) == Array(1,2)
res0: Boolean = false

However most of the other collections are comparable:

scala> List(1,2) == List(1,2)
res1: Boolean = true

Your options would be to either use one of the other collections (like a List) or to use deep for comparisons:

scala> Array(1,2).deep == Array(1,2).deep
res22: Boolean = true

scala> Array(1,2).deep == Array(1,3).deep
res23: Boolean = false
Comments