我在使用 scikit-learn 的 TfidfVectorizer 从文本数据中提取特征。我有一个 CSV 文件，包含评分（可以是 +1 或 -1）和评论（文本）。我将这些数据导入到一个 DataFrame 中，以便运行向量化器。
这是我的代码：
import pandas as pdimport numpy as npfrom sklearn.feature_extraction.text import TfidfVectorizerdf = pd.read_csv("train_new.csv", names = ['Score', 'Review'], sep=',')# x = df['Review'] == np.nan## print x.to_csv(path='FindNaN.csv', sep=',', na_rep = 'string', index=True)## print df.isnull().values.any()v = TfidfVectorizer(decode_error='replace', encoding='utf-8')x = v.fit_transform(df['Review'])这是我得到的错误的追溯信息：
Traceback (most recent call last): File "/home/PycharmProjects/Review/src/feature_extraction.py", line 16, in <module>x = v.fit_transform(df['Review']) File "/home/b/hw1/local/lib/python2.7/site- packages/sklearn/feature_extraction/text.py", line 1305, in fit_transform X = super(TfidfVectorizer, self).fit_transform(raw_documents) File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 817, in fit_transformself.fixed_vocabulary_) File "/home/b/work/local/lib/python2.7/site- packages/sklearn/feature_extraction/text.py", line 752, in _count_vocab for feature in analyze(doc): File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 238, in <lambda>tokenize(preprocess(self.decode(doc))), stop_words) File "/home/b/work/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 118, in decode raise ValueError("np.nan is an invalid document, expected byte or " ValueError: np.nan is an invalid document, expected byte or unicode string.我检查了 CSV 文件和 DataFrame，寻找任何被读取为 NaN 的内容，但没有找到任何东西。共有 18000 行，其中没有一行返回 isnan 为 True。
这是 df['Review'].head() 的样子：
0 This book is such a life saver. It has been s... 1 I bought this a few times for my older son and... 2 This is great for basics, but I wish the space... 3 This book is perfect! I'm a first time new mo... 4 During your postpartum stay at the hospital th... Name: Review, dtype: object回答：
你需要将 dtype object 转换为 unicode 字符串，正如追溯信息中明确提到的。
x = v.fit_transform(df['Review'].values.astype('U')) ## 甚至 astype(str) 也可以工作从 TFIDF Vectorizer 的文档页面：
fit_transform(raw_documents, y=None)
参数：raw_documents : 可迭代对象
一个可以生成 str、unicode 或 文件对象 的可迭代对象
