我在尝试使用 pyspark、CrossValidator 和 BinaryClassificationEvaluator 调优随机森林模型时遇到了一个错误。这是我的代码。
from pyspark.ml.evaluation import BinaryClassificationEvaluatorfrom pyspark.ml.classification import RandomForestClassifierfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml import Pipeline# Create a spark RandomForestClassifier using all default parameters.# Create a training, and testing dftraining_df, testing_df = raw_data_df.randomSplit([0.6, 0.4])# build a pipeline for analysisva = VectorAssembler().setInputCols(training_df.columns[0:110:]).setOutputCol('features')# featuresCol="features"rf = RandomForestClassifier(labelCol="quality")# Train the model and calculate the AUC using a BinaryClassificationEvaluatorrf_pipeline = Pipeline(stages=[va, rf]).fit(training_df)bce = BinaryClassificationEvaluator(labelCol="quality")# Check AUC before tuningbce.evaluate(rf_pipeline.transform(testing_df))from pyspark.ml.tuning import CrossValidator, ParamGridBuilderparamGrid = ParamGridBuilder().build()crossValidator = CrossValidator(estimator=rf_pipeline, estimatorParamMaps=paramGrid, evaluator=bce, numFolds=3)model = crossValidator.fit(training_df)它抛出了这个错误：
AttributeError: 'PipelineModel' object has no attribute 'fitMultiple'我该如何解决这个问题？
回答：
CrossValidator 的 estimator 需要一个 Pipeline 对象，而不是 Pipeline 模型。
请参考这个例子 – https://github.com/apache/spark/blob/master/examples/src/main/python/ml/cross_validator.py
你的代码应按以下方式修改
- 创建一个 pipeline
rf_pipe = Pipeline(stages=[va, rf])- 在 crossvalidator 中使用该 pipeline 作为 estimator
crossValidator = CrossValidator(estimator=rf_pipe, estimatorParamMaps=paramGrid, evaluator=bce, numFolds=3)总体来说 –
....# Train the model and calculate the AUC using a BinaryClassificationEvaluatorrf_pipe = Pipeline(stages=[va, rf])rf_pipeline = rf_pipe.fit(training_df)...crossValidator = CrossValidator(estimator=**rf_pipe**, estimatorParamMaps=paramGrid, evaluator=bce, numFolds=3)model = crossValidator.fit(training_df)