我有一段代码可以计算150个文档的TF-IDF矩阵。
import refrom sklearn.feature_extraction.text import TfidfVectorizerfrom nltk.corpus import stopwordsall_lines = []all_lines_corrected = []with open("Extracted Functional Goals - Stemmed.txt") as f: for line in f: temp = line.split(None,1) all_lines.append(temp[1])f.close()for a in range(len(all_lines)-1): all_lines_corrected.append(all_lines[a][:-2])all_lines_corrected.append(all_lines[len(all_lines)-1])stop_words = stopwords.words('english')tf = TfidfVectorizer(analyzer='word', stop_words = stop_words)tfidf_matrix = tf.fit_transform(all_lines_corrected).todense()query_string = raw_input("Enter string : ")
如何获取查询字符串的TF-IDF值?(我们可以假设它看起来像150个训练文档中的一个条目吗?)
回答:
你可以通过使用values = tf.transform([query_string])
来获取查询字符串的tf-idf值。结果将是一个稀疏矩阵,具有1行和N列,其中列是向量化器在训练文档中见过的N个唯一词的tfidf值。
类似于你的代码的简短示例:
from sklearn.feature_extraction.text import TfidfVectorizerall_lines = ["This is an example doc", "Another short example document .", "Just a third example"]tf = TfidfVectorizer(analyzer='word')tfidf_matrix = tf.fit_transform(all_lines)query_string = "This is a short example string"print "Query String:"print tf.transform([query_string])print "Example doc:"print tf.transform(["This is a short example doc"])
输出:
Query String: (0, 9) 0.546454011634 (0, 7) 0.546454011634 (0, 5) 0.546454011634 (0, 4) 0.32274454218Example doc: (0, 9) 0.479527938029 (0, 7) 0.479527938029 (0, 5) 0.479527938029 (0, 4) 0.283216924987 (0, 2) 0.479527938029