我想进行一个分类，其中一个文件完全用于训练，另一个文件完全用于测试。这可能吗？我尝试过以下方法：

import pandasimport numpy as npimport pandas as pdfrom sklearn import metricsfrom sklearn import cross_validationfrom sklearn.pipeline import Pipelinefrom sklearn.metrics import confusion_matrixfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import cross_val_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import precision_recall_fscore_support as scorefrom sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer, CountVectorizer, TfidfTransformerfrom sklearn.metrics import precision_score, recall_score, confusion_matrix, classification_report, accuracy_score, f1_score#csv file from traindf = pd.read_csv('data_train.csv', sep = ',')#csv file from testdf_test = pd.read_csv('data_test.csv', sep = ',')#Randomising the rows in the filedf = df.reindex(np.random.permutation(df.index))df_test = df_test.reindex(np.random.permutation(df_test.index))vect = CountVectorizer()X = vect.fit_transform(df['data_train'])y = df['label']X_T = vect.fit_transform(df_test['data_test'])y_t = df_test['label']X_train, y_train = train_test_split(X, y, test_size = 0, random_state = 100)X_test, y_test = train_test_split(X_T, y_t, test_size = 0, random_state = 100)tf_transformer = TfidfTransformer(use_idf=False).fit(X) X_train_tf = tf_transformer.transform(X) X_train_tf.shapetfidf_transformer = TfidfTransformer()X_train_tfidf = tfidf_transformer.fit_transform(X) X_train_tfidf.shapetf_transformer = TfidfTransformer(use_idf=False).fit(X_T) X_train_tf_teste = tf_transformer.transform(X_T) X_train_tf_teste.shapetfidf_transformer = TfidfTransformer()X_train_tfidf_teste = tfidf_transformer.fit_transform(X_T) X_train_tfidf_teste.shape#RegLogclf = LogisticRegression().fit(X_train, y_train)y_pred = clf.predict(X_test)print("confusion matrix")print(confusion_matrix(y_test, y_pred, labels = y))print("F-score")print(f1_score(y_test, y_pred, average=None))print(precision_score(y_test, y_pred, average=None))print(recall_score(y_test, y_pred, average=None)) print("cross validation")scores = cross_validation.cross_val_score(clf, X, y, cv = 10)print(scores)print("Accuracy: {} +/- {}".format(scores.mean(), scores.std() * 2))

我将test_size设置为零，因为我不希望对这些文件进行分割。我还在训练和测试文件上应用了Count和TFIDF方法。

我的输出错误：

Traceback (most recent call last):

File “classif.py”, line 34, in X_train, y_train = train_test_split(X, y, test_size = 0, random_state = 100)

ValueError: too many values to unpack (expected 2)

回答：

你在train_test_split中遇到的错误已经被@Alexis明确指出并解决了。我也再次建议不要使用train_test_split，因为它除了打乱顺序外不会做其他事情，而你已经完成了打乱操作。

但我想强调另一个重要点，即如果你将训练和测试文件分开保存，那么请不要分别拟合向量化器。这会导致训练和测试文件的列不同。例如：

cv = CountVectorizer()train=['Hi this is stack overflow']cv.fit(train)cv.get_feature_names()

输出： ['hi', 'is', 'overflow', 'stack', 'this']

test=['Hi that is not stack overflow']cv.fit(test)cv.get_feature_names()

输出： ['hi', 'is', 'not', 'overflow', 'stack', 'that']

因此，分别拟合会导致列不匹配。所以，你应该首先合并训练和测试文件，然后集体拟合变换向量化器，或者如果你事先没有测试数据，你可以仅使用在训练数据上拟合的向量化器来变换测试数据，这将忽略训练数据中不存在的词语。