我对机器学习和Spark ML都比较新手，正在尝试使用Spark ML中的神经网络构建预测模型，但在对我的学习模型调用.transform方法时遇到了这个错误。问题是由OneHotEncoder的使用引起的，因为如果不使用它，一切都能正常工作。我已经尝试从管道中移除OneHotEncoder。
我的问题是：如何使用OneHotEncoder而不产生这个错误？
java.lang.IllegalArgumentException: requirement failed: A & B Dimension mismatch! at scala.Predef$.require(Predef.scala:224) at org.apache.spark.ml.ann.BreezeUtil$.dgemm(BreezeUtil.scala:41) at org.apache.spark.ml.ann.AffineLayerModel.eval(Layer.scala:163) at org.apache.spark.ml.ann.FeedForwardModel.forward(Layer.scala:482) at org.apache.spark.ml.ann.FeedForwardModel.predict(Layer.scala:529)我的代码如下：
test_pandas_df = pd.read_csv( '/home/piotrek/ml/adults/adult.test', names=header, skipinitialspace=True)train_pandas_df = pd.read_csv( '/home/piotrek/ml/adults/adult.data', names=header, skipinitialspace=True)train_df = sqlContext.createDataFrame(train_pandas_df)test_df = sqlContext.createDataFrame(test_pandas_df)joined = train_df.union(test_df)assembler = VectorAssembler().setInputCols(features).setOutputCol("features")label_indexer = StringIndexer().setInputCol( "label").setOutputCol("label_index")label_indexer_fit = [label_indexer.fit(joined)]string_indexers = [StringIndexer().setInputCol( name).setOutputCol(name + "_index").fit(joined) for name in categorical_feats]one_hot_pipeline = Pipeline().setStages([OneHotEncoder().setInputCol( name + '_index').setOutputCol(name + '_one_hot') for name in categorical_feats])mlp = MultilayerPerceptronClassifier().setLabelCol(label_indexer.getOutputCol()).setFeaturesCol( assembler.getOutputCol()).setLayers([len(features), 20, 10, 2]).setSeed(42L).setBlockSize(1000).setMaxIter(500)pipeline = Pipeline().setStages(label_indexer_fit + string_indexers + [one_hot_pipeline] + [assembler, mlp])model = pipeline.fit(train_df)# compute accuracy on the test setresult = model.transform(test_df)## FAILS ON RESULTpredictionAndLabels = result.select("prediction", "label_index")evaluator = MulticlassClassificationEvaluator(labelCol="label_index")print "-------------------------------"print("Test set accuracy = " + str(evaluator.evaluate(predictionAndLabels)))print "-------------------------------"谢谢！
回答：
你的模型中的layers Param 设置不正确：
setLayers([len(features), 20, 10, 2])第一层应该反映输入特征的数量，这通常与编码前的原始列数不同。
如果你事先不知道特征的总数，你可以例如分开进行特征提取和模型训练。伪代码如下：
feature_pipeline_model = (Pipeline() .setStages(...) # 仅特征提取 .fit(train_df))train_df_features = feature_pipeline_model.transform(train_df)layers = [ train_df_features.schema["features"].metadata["ml_attr"]["num_attrs"], 20, 10, 2]