jmbeleta jmbeleta - 2 months ago 6
Scala Question

ScalaCheck, different behaviour when calling Test.check or Test.checkProperties

I have the following code that uses ScalaCheck properties to test some class

package quickcheck.tests

import org.scalacheck.Arbitrary._
import org.scalacheck.Gen._
import org.scalacheck.Prop._
import org.scalacheck._
import org.scalacheck.util.ConsoleReporter

class Heap
{
}

object Heap
{
val empty = new Heap
def insert(i: Int, h: Heap) = new Heap
}

class TestCheck extends Properties("Heap")
{
lazy val genHeap: Gen[Heap] =
{
def sizedHeap(size: Int): Gen[Heap] =
{
if (size <= 0)
Heap.empty
else
for
(
i <- arbitrary[Int] suchThat(_ > Int.MinValue);
s <- choose(0, size);
h <- sizedHeap(s)
)
yield
Heap.insert(i, h)
}

Gen.sized[Heap](sizedHeap)
}

implicit lazy val arbHeap: Arbitrary[Heap] = Arbitrary(genHeap)

property("test1") = forAll
{
(h: Heap) => true
}

property("test2") = forAll
{
(h1: Heap, h2: Heap, n: Int) => true
}
}

object MyTest extends App
{
println("*** TEST 1")
val checkHeap = new TestCheck
Test.checkProperties(Test.Parameters.default.withTestCallback(ConsoleReporter(1)),
checkHeap)

println("*** TEST 2")
val checkHeap2 = new TestCheck
checkHeap2.check

println("*** TEST 3")
val checkHeap3 = new TestCheck
Test.check(Test.Parameters.default.withTestCallback(ConsoleReporter(1)), checkHeap)
}


If I run it thru the ScalaCheck
Test
class I get different results if I use method
Test.checkProperties
or method
Test.check
.

This is the output I get:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
! Gave up after only 40 passed tests. 202 tests were discarded.


My questions is why TEST1 gives different result than TEST3.

If I remove the
suchThat
filter and leave the
for
statement in the
sizeHead
method as this:

for
(
i <- arbitrary[Int]
s <- choose(0, size);
h <- sizedHeap(s)
)
yield
Heap.insert(i, h)


I get the following result:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
+ OK, passed 100 tests.


Is this a bug or it is the correct behaviour? Keep in mind that ScalaTest
Checkers.check
uses
Test.check
.

Answer

You get different results because you're doing different things. Your first two tests—which are effectively the same—check all properties separately, whereas your third one tests all properties as if they were a single property.

Take a look at the signatures of Test.check and Test.checkProperties: The former takes a single Prop, whereas the latter takes a Properties container.

In scalacheck 1.12 Properties inherits from Prop; if you pass Properties as Prop you get a new property that tests whether all contained properties hold. The consequence is that you test all your TestCheck properties with a single check configuration (i.e. generators, iteration limits, etc.), and naturally that check configuration is exhausted at some point.

So yes, that's expected behaviour. Of course, it's absolutely confusing, hence scalacheck 1.13 removed this feature: Properties does not inherit from Prop anymore; your example would not compile on scalacheck 1.13 anymore.

Comments