我在尝试以下代码

from sklearn.feature_extraction.text import TfidfVectorizerimport numpy as nptrain_data = ["football is the sport","gravity is the movie", "education is imporatant"]vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,                                                 stop_words='english')print "Applying first train data"X_train = vectorizer.fit_transform(train_data)print vectorizer.get_feature_names()print "\n\nApplying second train data"train_data = ["cricket", "Transformers is a film","AIMS is a college"]X_train = vectorizer.transform(train_data)print vectorizer.get_feature_names()print "\n\nApplying fit transform onto second train data"X_train = vectorizer.fit_transform(train_data)print vectorizer.get_feature_names()

这个的输出是

Applying first train data[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']Applying second train data[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport'] Applying fit transform onto second train data[u'aims', u'college', u'cricket', u'film', u'transformers']

我首先使用fit_transform方法将第一组数据应用到vectorizer上，因此它返回了如[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']这样的特征名称。之后，我将另一组训练数据应用到同一个vectorizer上，但由于没有使用fit或fit_transform方法，它返回了相同的特征名称。但是我想知道如何在不覆盖之前的特征的情况下更新vectorizer的特征。如果我再次使用fit_transform，之前的特征就会被覆盖。因此，我想更新vectorizer的特征列表。我希望得到类似于[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport',u'aims', u'college', u'cricket', u'film', u'transformers']的结果。如何实现这一点呢？

回答：

在scikit-learn的术语中，这被称为部分拟合（partial fit），而你无法用TfidfVectorizer来实现这一点。有两种解决方法：

• 将两个训练集连接起来并重新向量化
• 使用支持部分拟合的HashingVectorizer。然而，由于它对特征进行哈希处理，因此没有get_feature_names方法，无法保留原始特征。另一个优点是这种方法在内存使用上更加高效。

第一种方法的示例：

from sklearn.feature_extraction.text import TfidfVectorizerimport numpy as nptrain_data1 = ["football is the sport", "gravity is the movie", "education is important"]vectorizer = TfidfVectorizer(stop_words='english')print("Applying first train data")X_train = vectorizer.fit_transform(train_data1)print(vectorizer.get_feature_names())print("\n\nApplying second train data")train_data2 = ["cricket", "Transformers is a film", "AIMS is a college"]X_train = vectorizer.transform(train_data2)print(vectorizer.get_feature_names())print("\n\nApplying fit transform onto second train data")X_train = vectorizer.fit_transform(train_data1 + train_data2)print(vectorizer.get_feature_names())

输出：

Applying first train data['education', 'football', 'gravity', 'important', 'movie', 'sport']Applying second train data['education', 'football', 'gravity', 'important', 'movie', 'sport']Applying fit transform onto second train data['aims', 'college', 'cricket', 'education', 'film', 'football', 'gravity', 'important', 'movie', 'sport', 'transformers']