我正在尝试从TF-IDF结果向量中获取分数数组。例如：
rescaledData.select("words", "features").show()+-----------------------------+---------------------------------------------------------------------------------------------+|words |features |+-----------------------------+---------------------------------------------------------------------------------------------+|[a, b, c] |(4527,[0,1,31],[0.6363067860791387,1.0888040725098247,4.371858972705023]) ||[d] |(4527,[8],[2.729945780576634]) |+-----------------------------+---------------------------------------------------------------------------------------------+rescaledData.select(rescaledData['features'].getItem('values')).show()但我得到的不是数组，而是一个错误。
AnalysisException: u"Can't extract value from features#1786: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;"我想要的结果是
+--------------------------+-----------------------------------------------------------+|words |features |+--------------------------+-----------------------------------------------------------+|[a, b, c] |[0.6363067860791387, 1.0888040725098247, 4.371858972705023]|+--------------------------+-----------------------------------------------------------+如何解决这个问题？
回答：
另一种方法是创建一个udf来从稀疏向量中获取值：
from pyspark.sql.functions import udffrom pyspark.sql.types import DoubleType, ArrayTypesparse_values = udf(lambda v: v.values.tolist(), ArrayType(DoubleType()))df.withColumn("features", sparse_values("features")).show(truncate=False)+---------+-----------------------------------------------------------+|word |features |+---------+-----------------------------------------------------------+|[a, b, c]|[0.6363067860791387, 1.0888040725098247, 4.371858972705023]||[d] |[2.729945780576634] |+---------+-----------------------------------------------------------+