我试图将从Spark ML库创建的模型对象保存起来。

然而，这引发了一个错误：

线程 “main” 中的异常 java.lang.NoSuchMethodError: org.apache.spark.ml.PipelineModel.save(Ljava/lang/String;)V at com.sf.prediction$.main(prediction.scala:61) at com.sf.prediction.main(prediction.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

以下是我的依赖项：

    <dependency>        <groupId>org.scalatest</groupId>        <artifactId>scalatest_2.10</artifactId>        <version>2.1.7</version>        <scope>test</scope>    </dependency>    <dependency>        <groupId>org.apache.maven.plugins</groupId>        <artifactId>maven-shade-plugin</artifactId>        <version>2.4.3</version>        <type>maven-plugin</type>    </dependency>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-core_2.10</artifactId>        <version>1.6.0</version>    </dependency>    <dependency>        <groupId>org.scala-lang</groupId>        <artifactId>scala-parser-combinators</artifactId>        <version>2.11.0-M4</version>    </dependency>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-sql_2.10</artifactId>        <version>1.6.0</version>    </dependency>    <dependency>        <groupId>org.apache.commons</groupId>        <artifactId>commons-csv</artifactId>        <version>1.2</version>    </dependency>    <dependency>        <groupId>com.databricks</groupId>        <artifactId>spark-csv_2.10</artifactId>        <version>1.4.0</version>    </dependency>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-hive_2.10</artifactId>        <version>1.6.1</version>    </dependency>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-mllib_2.10</artifactId>        <version>1.6.0</version>    </dependency>

我还想将模型生成的数据框保存为CSV格式。

model.transform(df).select("features","label","prediction").show()import org.apache.spark.SparkConfimport org.apache.spark.SparkContextimport org.apache.spark.sql.SQLContextimport org.apache.spark.sql.functions._import org.apache.spark.SparkConfimport org.apache.spark.sql.hive.HiveContextimport org.apache.spark.ml.feature.OneHotEncoderimport org.apache.spark.ml.feature.VectorAssemblerimport org.apache.spark.ml.classification.LogisticRegressionimport org.apache.spark.ml.Pipelineimport org.apache.spark.ml.PipelineModel._import org.apache.spark.ml.feature.{IndexToString, StringIndexer, VectorIndexer}import org.apache.spark.ml.util.MLWritableobject prediction {  def main(args: Array[String]): Unit = {     val conf = new SparkConf()             .setMaster("local[2]")             .setAppName("conversion")    val sc = new SparkContext(conf)    val hiveContext = new HiveContext(sc)    val df = hiveContext.sql("select * from prediction_test")    df.show()    val credit_indexer = new StringIndexer().setInputCol("transaction_credit_card").setOutputCol("creditCardIndex").fit(df)    val category_indexer = new StringIndexer().setInputCol("transaction_category").setOutputCol("categoryIndex").fit(df)    val location_flag_indexer = new StringIndexer().setInputCol("location_flag").setOutputCol("locationIndex").fit(df)    val label_indexer = new StringIndexer().setInputCol("fraud").setOutputCol("label").fit(df)    val assembler =  new VectorAssembler().setInputCols(Array("transaction_amount", "creditCardIndex","categoryIndex","locationIndex")).setOutputCol("features")    val lr = new LogisticRegression().setMaxIter(10).setRegParam(0.01)    val pipeline = new Pipeline().setStages(Array(credit_indexer, category_indexer, location_flag_indexer, label_indexer, assembler, lr))    val model = pipeline.fit(df)    pipeline.save("/user/f42h/prediction/pipeline")    model.save("/user/f42h/prediction/model") //   val sameModel = PipelineModel.load("/user/bob/prediction/model")    model.transform(df).select("features","label","prediction")  }}

回答：

您使用的是Spark 1.6.0，据我所知，ML模型的保存和加载功能是从2.0版本开始才可用的。您可以使用带有2.0.0-preview版本的工件进行预览： http://search.maven.org/#search%7Cga%7C1%7Cg%3Aorg.apache.spark%20v%3A2.0.0-preview