GwydionFR GwydionFR - 8 months ago 85
Python Question

Pyspark - Get all parameters of models created with ParamGridBuilder

Im using pySpark 2.0 for a kaggle competition. I'd like to know the behavior of a model (randomForest) depending on different parameters. ParamGridBuilder() allows to specify different values for a single parameters, and then perform (i guess) a cartesian product of the entire set of parameters. Assuming my dataframe is already defined:

rdc = RandomForestClassifier()
pipeline = Pipeline(stages=STAGES + [rdc])
paramGrid = ParamGridBuilder().addGrid(rdc.maxDepth, [3, 10, 20])
.addGrid(rdc.minInfoGain, [0.01, 0.001])
.addGrid(rdc.numTrees, [5, 10, 20, 30])
evaluator = MulticlassClassificationEvaluator()
valid = TrainValidationSplit(estimator=pipeline,
model =
result = model.bestModel.transform(df)

Ok so now I'm able to retrieves simple information with a handmade function:

def evaluate(result):
predictionAndLabels ="prediction", "label")
metrics = ["f1","weightedPrecision","weightedRecall","accuracy"]
for m in metrics:
evaluator = MulticlassClassificationEvaluator(metricName=m)
print(str(m) + ": " + str(evaluator.evaluate(predictionAndLabels)))

Now I want several things:

*What are the parameters of the best model? This post partially answers the question: How to extract model hyper-parameters from in PySpark?

*What are the parameters of all models ?
*What are the results (aka recall, accuracy, etc...) of each model ? I only found
that displays (it seems) a list containing the accuracy of each model, but I can't get to know which model to refers

If I can retrieve all those informations, I should be able to display graphs, bar charts, and work as I do with panda and sklearn.


Long story short you simply cannot get parameters for all models because, similarly to CrossValidator, TrainValidationSplitModel retains only the best model. These classes are designed for semi-automated model selection not exploration or experiments.

What are the parameters of all models?

While you cannot retrieve actual models validationMetrics correspond to input Params so you should be able to simply zip both:

from typing import Dict, Tuple, List, Any
from import Param
from import TrainValidationSplitModel

EvalParam = List[Tuple[float, Dict[Param, Any]]]

def get_metrics_and_params(model: TrainValidationSplitModel) -> EvalParam:
    return list(zip(model.validationMetrics, model.getEstimatorParamMaps()))

to get some about relationship between metrics and parameters.

If you need more information you should use Pipeline Params. It will preserve all model which can be used for further processing:

models =, params=paramGrid)