根据PySpark的文档,fmeasure()
函数接受两个参数,即label
和beta
:
fMeasure(label=None, beta=None)
这里的beta是什么意思?
我使用了一个非常简单的RDD数据集(虽然它是数据框格式,但我已转换为RDD):
+----------+-----+|prediction|label|+----------+-----+| 1| 1|| 1| 1|| 0| 1|| 0| 0|| 1| 0|| 1| 0|| 0| 0|| 0| 0|| 1| 1|| 1| 1|+----------+-----+
当我运行以下命令时:
multi_metrics = MulticlassMetrics(rdd)print 'fMeasure: ', multi_metrics.fMeasure(1)
我得到了这个错误:
print 'fMeasure: ', multi_metrics.fMeasure(1) File "/usr/hdp/current/spark-client/python/pyspark/mllib/evaluation.py", line 259, in fMeasure return self.call("fMeasure", label) File "/usr/hdp/current/spark-client/python/pyspark/mllib/common.py", line 146, in call return callJavaFunc(self._sc, getattr(self._java_model, name), *a) File "/usr/hdp/current/spark-client/python/pyspark/mllib/common.py", line 123, in callJavaFunc return _java2py(sc, func(*args)) File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/usr/hdp/current/spark-client/python/pyspark/sql/utils.py", line 45, in deco return f(*a, **kw) File "/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 312, in get_return_value format(target_id, ".", name, value))Py4JError: An error occurred while calling o154.fMeasure. Trace:py4j.Py4JException: Method fMeasure([class java.lang.Integer]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344) at py4j.Gateway.invoke(Gateway.java:252) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745)
回答:
这里的beta是什么意思?
Spark的MulticlassMetrics
实现了$F_{\beta}$度量,当$\beta$设为1时,它与传统的F度量一致。$\beta$参数允许控制F度量中精确度与召回率的相对贡献。
关于错误:如果你查看实现,你会发现它实际上期望一个Double
类型。这里是PySpark中fMeasure方法的包装器,以及实际的实现(用Scala编写)。
所以你应该像这样调用它,例如:
multi_metrics = MulticlassMetrics(rdd)print 'fMeasure: ', multi_metrics.fMeasure(1.0,1.0)