Dragonly Dragonly - 3 months ago 10
Scala Question

Where's Product related statistics in TreeNode class in Spark 2.0.0?


I'm trying to get a grasp of the

org.apache.spark.sql.catalyst.trees.TreeNode


class of Spark 2.0.0 source code, but found the Product part funny.

the definition begins as

abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {


then I inspected
trait Product
and could not figure out how does
TreeNode
get to know the statistics which method
ProductArity
or
ProductElement
of
trait Product
need.

I searched for a while and find out somebody pointing out that the scala compiler will figure out those statistics for
case class
, but the
TreeNode
is not a case class and I cannot find some protocol or hints that
Product
can get from the implementing class to get the needed numbers such as
ProductArity
returns.

So does anyone know what's behind the black magic?

Answer

TreeNode is not a case class

But every non-abstract class extending TreeNode is. There could also be non-case classes which implement productArity and productElement manually, but I don't think there are any in Spark.