我来自R语言背景，所以scikit的API对我来说仍然很 confusing。我在按照这个教程 http://michelleful.github.io/code-blog/2015/06/20/pipelines/ 学习关于Pipelines的内容。所以让我们创建一个假数据集仅供参考：

x1,x2,yfoo,zoo,1bar,moo,2goo,too,3roo,zoo,4too,moo,5

我的目标非常简单：使用x1和x2的单独tfidf矩阵，加上从x1和x2中提取的一些自定义特征（例如，词长度等）来训练y的线性回归模型。

让我们从仅使用x1的tfidf开始这个较简单的任务。这里是完整的代码：

from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_log_errorfrom sklearn.linear_model import LinearRegressionfrom sklearn.pipeline import Pipeline, FeatureUnionfrom sklearn.model_selection import KFoldfrom sklearn.model_selection import cross_val_scorefrom sklearn.metrics import fbeta_score, make_scorerfrom sklearn.base import BaseEstimator, TransformerMixinimport pandas as pdimport numpy as npimport timeimport reimport mathdef clip_RMSLE(y, y_pred, **kwargs):    y_pred[y_pred < 0] = 0.0    to_sum = [(math.log(y_pred[i] + 1) - math.log(y[i] + 1)) ** 2.0 for i,pred in enumerate(y_pred)]    return (sum(to_sum) * (1.0/len(y))) ** 0.5class ColumnNgram(BaseEstimator, TransformerMixin):    def __init__(self, colname, tokenizer, ngram_rg):        self.colname = colname        self.tokenizer = tokenizer        self.ngram_rg = ngram_rg        self.tfidf = None    def transform(self, df, y=None):         tfidf = TfidfVectorizer(tokenizer=self.tokenizer, ngram_range=self.ngram_rg)         return tfidf.fit_transform(df[self.colname].values)    def fit(self, df, y=None):        return selfstart = time.time()seed = 1991ngram_rg = (1,2)RMSLE = make_scorer(clip_RMSLE, greater_is_better=False)def tokenizer(text):    if text:        result = re.findall('[a-z]{2,}', text.lower())    else:        result = []    return resultdf = pd.read_csv('fake.csv', sep=',')y = df['y'].valuespipeline = Pipeline([('tfidf', ColumnNgram('x1', tokenizer, ngram_rg)),('linear_reg', LinearRegression(n_jobs=1))])kfold = KFold(n_splits=2, random_state=seed)results = cross_val_score(pipeline, df, y, cv=kfold, scoring=RMSLE)print(results)print(results.mean())end = time.time()print('Timeto finish this thing: %0.2fs' % (end - start))

我遇到了错误 ValueError: dimension mismatch，可能是因为某些术语不会同时出现在训练/验证折叠中。正确的做法是什么？谢谢！

回答：

将你的ColumnNgram改成这样：

class ColumnNgram(BaseEstimator, TransformerMixin):    def __init__(self, colname, tokenizer, ngram_rg):        self.colname = colname        self.tokenizer = tokenizer        self.ngram_rg = ngram_rg        self.tfidf = None    def transform(self, df, y=None):        return self.tfidf.transform(df[self.colname].values)    def fit(self, df, y=None):        self.tfidf = TfidfVectorizer(tokenizer=self.tokenizer, ngram_range=self.ngram_rg)        self.tfidf.fit(df[self.colname].values)        return self

你应该在fit()中声明并学习训练数据。目前你在每次调用transform()时都在重新拟合数据，这显然会导致训练和验证集中的特征不同，正如你所建议的。

正确的方法是保持一个TfidfVectorizer，它在fit()期间学习数据，然后在transform()中仅转换新数据，而不是重新拟合新数据。