GwydionFR GwydionFR - 2 months ago 38
Python Question

Pyspark - Get all parameters of models created with ParamGridBuilder

Im using pySpark 2.0 for a kaggle competition. I'd like to know the behavior of a model (randomForest) depending on different parameters. ParamGridBuilder() allows to specify different values for a single parameters, and then perform (i guess) a cartesian product of the entire set of parameters. Assuming my dataframe is already defined:

rdc = RandomForestClassifier()
pipeline = Pipeline(stages=STAGES + [rdc])
paramGrid = ParamGridBuilder().addGrid(rdc.maxDepth, [3, 10, 20])
.addGrid(rdc.minInfoGain, [0.01, 0.001])
.addGrid(rdc.numTrees, [5, 10, 20, 30])
.build()
evaluator = MulticlassClassificationEvaluator()
valid = TrainValidationSplit(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=evaluator,
trainRatio=0.50)
model = valid.fit(df)
result = model.bestModel.transform(df)


Ok so now I'm able to retrieves simple information with a handmade function:

def evaluate(result):
predictionAndLabels = result.select("prediction", "label")
metrics = ["f1","weightedPrecision","weightedRecall","accuracy"]
for m in metrics:
evaluator = MulticlassClassificationEvaluator(metricName=m)
print(str(m) + ": " + str(evaluator.evaluate(predictionAndLabels)))


Now I want several things:

*What are the parameters of the best model? This post partially answers the question: How to extract model hyper-parameters from spark.ml in PySpark?

*What are the parameters of all models ?
*What are the results (aka recall, accuracy, etc...) of each model ? I only found
print(model.validationMetrics)
that displays (it seems) a list containing the accuracy of each model, but I can't get to know which model to refers

If I can retrieve all those informations, I should be able to display graphs, bar charts, and work as I do with panda and sklearn.

Answer

Long story short you simply cannot get parameters for all models because, similarly to CrossValidator, TrainValidationSplitModel retains only the best model. These classes are designed for semi-automated model selection not exploration or experiments.

What are the parameters of all models?

While you cannot retrieve actual models validationMetrics correspond to input Params so you should be able to simply zip both:

from typing import Dict, Tuple, List, Any
from pyspark.ml.param import Param
from pyspark.ml.tuning import TrainValidationSplitModel

EvalParam = List[Tuple[float, Dict[Param, Any]]]

def get_metrics_and_params(model: TrainValidationSplitModel) -> EvalParam:
    return list(zip(model.validationMetrics, model.getEstimatorParamMaps()))

to get some about relationship between metrics and parameters.

If you need more information you should use Pipeline Params. It will preserve all model which can be used for further processing:

models = pipeline.fit(df, params=paramGrid)