Georg Heiler Georg Heiler - 5 days ago 5
Scala Question

Spark custom estimator override Nothing?

I am implementing a custom estimator which has troubles accessing a parameter

ava.util.NoSuchElementException: Failed to find a default value for isInList
. It is defined as follows:

trait PreprocessingParams extends Params {
final val isInList = new Param[Array[String]](this, "isInList", "list of isInList items")
}


To better debug the problem I created a minimal example here https://gist.github.com/geoHeil/8dc7a6b6938a517f068e7fd6a981ed12 the
ExampleTrans
works just fine. However, I would rather like to include the functionality of the transformer into an estimator which performs some data cleaning as well.

But now I face strange compile issues
overriding method has wrong type - expecting Nothing


What is wrong with the return types of my
ExampleEstimator
?

Answer

You didn't put generic types.

Use:

class ExampleEstimator(override val uid: String) extends Estimator[ExampleTransModel] with PreprocessingParams {

Estimator is parametrized with generic type M that extends Model. fit() should return type M. Without specifying explicity, Nothing is used

Comments