我在进行一个模糊匹配项目时，发现了一个非常有趣的方法：awesome_cossim_top
我大致理解了这个方法的定义，但不明白当我们使用fit_transform时发生了什么
import pandas as pdimport sqlite3 as sqlfrom sklearn.feature_extraction.text import TfidfVectorizerimport numpy as npfrom scipy.sparse import csr_matriximport sparse_dot_topn.sparse_dot_topn as ctimport redef ngrams(string, n=3): string = re.sub(r'[,-./]|\sBD',r'', re.sub(' +', ' ',str(string))) ngrams = zip(*[string[i:] for i in range(n)]) return [''.join(ngram) for ngram in ngrams]def awesome_cossim_top(A, B, ntop, lower_bound=0): # force A and B as a CSR matrix. # If they have already been CSR, there is no overhead A = A.tocsr() B = B.tocsr() M, _ = A.shape _, N = B.shape idx_dtype = np.int32 nnz_max = M*ntop indptr = np.zeros(M+1, dtype=idx_dtype) indices = np.zeros(nnz_max, dtype=idx_dtype) data = np.zeros(nnz_max, dtype=A.dtype) ct.sparse_dot_topn( M, N, np.asarray(A.indptr, dtype=idx_dtype), np.asarray(A.indices, dtype=idx_dtype), A.data, np.asarray(B.indptr, dtype=idx_dtype), np.asarray(B.indices, dtype=idx_dtype), B.data, ntop, lower_bound, indptr, indices, data) print('ct.sparse_dot_topn: ', ct.sparse_dot_topn) return csr_matrix((data,indices,indptr),shape=(M,N)) def get_matches_df(sparse_matrix, A, B, top=100): non_zeros = sparse_matrix.nonzero() sparserows = non_zeros[0] sparsecols = non_zeros[1] if top: nr_matches = top else: nr_matches = sparsecols.size left_side = np.empty([nr_matches], dtype=object) right_side = np.empty([nr_matches], dtype=object) similairity = np.zeros(nr_matches) for index in range(0, nr_matches): left_side[index] = A[sparserows[index]] right_side[index] = B[sparsecols[index]] similairity[index] = sparse_matrix.data[index] return pd.DataFrame({'left_side': left_side, 'right_side': right_side, 'similairity': similairity})以下是我感到困惑的脚本：为什么我们应该先使用fit_transform，然后再在同一个向量化器上仅使用transform。我尝试打印了向量化器和矩阵的一些输出，比如print(vectorizer.get_feature_names())，但不明白其中的逻辑。
有谁能帮我澄清吗？
非常感谢！！
Col_clean = 'fruits_normalized'Col_dirty = 'fruits'#read tabledata_dirty={f'{Col_dirty}':['I am an apple', 'You are an apple', 'Aple', 'Appls', 'Apples']}data_clean= {f'{Col_clean}':['apple', 'pear', 'banana', 'apricot', 'pineapple']}df_clean = pd.DataFrame(data_clean)df_dirty = pd.DataFrame(data_dirty)Name_clean = df_clean[f'{Col_clean}'].unique()Name_dirty= df_dirty[f'{Col_dirty}'].unique()vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)clean_idf_matrix = vectorizer.fit_transform(Name_clean)dirty_idf_matrix = vectorizer.transform(Name_dirty)matches = awesome_cossim_top(dirty_idf_matrix, clean_idf_matrix.transpose(),1,0)matches_df = get_matches_df(matches, Name_dirty, Name_clean, top = 0)with pd.option_context('display.max_rows', None, 'display.max_columns', None): matches_df.to_excel("output_apple.xlsx")print('done')回答：
TfidfVectorizer.fit_transform用于从训练数据集中创建词汇表，而TfidfVectorizer.transform用于将该词汇表映射到测试数据集，以便测试数据中的特征数量与训练数据保持一致。以下示例可能会有所帮助：
import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizer创建一个虚拟的训练数据：
train = pd.DataFrame({'Text' :['I am a data scientist','Cricket is my favorite sport', 'I work on Python regularly', 'Python is very fast for data mining', 'I love playing cricket'], 'Category' :['Data_Science','Cricket','Data_Science','Data_Science','Cricket']})以及一个小型的测试数据：
test = pd.DataFrame({'Text' :['I am new to data science field', 'I play cricket on weekends', 'I like writing Python codes'], 'Category' :['Data_Science','Cricket','Data_Science']})创建一个TfidfVectorizer()对象，称为vectorizer
vectorizer = TfidfVectorizer()在训练数据上拟合它
X_train = vectorizer.fit_transform(train['Text'])print(vectorizer.get_feature_names())#['am', 'cricket', 'data', 'fast', 'favorite', 'for', 'is', 'love', 'mining', 'my', 'on', 'playing', 'python', 'regularly', 'scientist', 'sport', 'very', 'work']feature_names = vectorizer.get_feature_names()df= pd.DataFrame(X.toarray(),columns=feature_names)现在看看在测试数据集上做同样的事情会发生什么：
vectorizer_test = TfidfVectorizer()X_test = vectorizer_test.fit_transform(test['Text'])print(vectorizer_test.get_feature_names())#['am', 'codes', 'cricket', 'data', 'field', 'like', 'new', 'on', 'play', 'python', 'science', 'to', 'weekends', 'writing']feature_names_test = vectorizer_test.get_feature_names()df_test= pd.DataFrame(X_test.toarray(),columns = feature_names_test)它在测试数据集上创建了另一个词汇表，相比训练数据的18个单词（列），它有14个独特的单词（列）。
现在，如果您在训练数据上训练一个用于文本分类的机器学习算法，并尝试在测试数据的矩阵上进行预测，它将失败并生成一个错误，指出训练和测试数据之间的特征不同。
为了克服这个错误，我们在文本分类中会做如下操作：
X_test_from_train = vectorizer.transform(test['Text'])feature_names_test_from_train = vectorizer.get_feature_names()df_test_from_train = pd.DataFrame(X_test_from_train.toarray(),columns = feature_names_test_from_train)在这里您可能会注意到，我们没有使用fit_transform命令，而是对测试数据使用了transform，原因是一样的，即在对测试数据进行预测时，我们只希望使用训练和测试数据中都存在的特征，以避免特征不匹配错误。
希望这对您有所帮助！！
