我已经开发了一个用于多标签分类的文本模型。使用 OneVsRestClassifier 的 LinearSVC 模型利用 sklearn 的 Pipeline 和 FeatureUnion 进行模型准备。
主要输入特征包括一个名为 response 的文本列，以及从之前的 LDA 主题模型生成的 5 个主题概率（称为 t1_prob 到 t5_prob），用于预测 5 个可能的标签。管道中还有其他特征创建步骤，用于生成 TfidfVectorizer。
我最终使用 ItemSelector 调用每个列，并对这些主题概率列单独执行了 5 次 ArrayCaster（请参阅下面的代码以了解函数定义）。有没有更好的方法使用 FeatureUnion 在管道中选择多个列？（这样我就不必重复操作 5 次）
我想知道是否有必要重复 topic1_feature 到 topic5_feature 的代码，或者是否有更简洁的方法选择多个列？
我输入的数据是一个 Pandas DataFrame：
id response label_1 label_2 label3 label_4 label_5 t1_prob t2_prob t3_prob t4_prob t5_prob1 Text from response... 0.0 0.0 0.0 0.0 0.0 0.0 0.0625 0.0625 0.1875 0.0625 0.12502 Text to model with... 0.0 0.0 0.0 0.0 0.0 0.0 0.1333 0.1333 0.0667 0.0667 0.0667 3 Text to work with ... 0.0 0.0 0.0 0.0 0.0 0.0 0.1111 0.0938 0.0393 0.0198 0.2759 4 Free text comments ... 0.0 0.0 1.0 1.0 0.0 0.0 0.2162 0.1104 0.0341 0.0847 0.0559 x_train 是 response 和 5 个主题概率列（t1_prob, t2_prob, t3_prob, t4_prob, t5_prob）。
y_train 是 5 个 label 列，我对它们调用了 .values 以返回 DataFrame 的 numpy 表示。（label_1, label_2, label3, label_4, label_5）
样本 DataFrame：
import pandas as pdcolumn_headers = ["id", "response", "label_1", "label_2", "label3", "label_4", "label_5", "t1_prob", "t2_prob", "t3_prob", "t4_prob", "t5_prob"]input_data = [ [1, "Text from response",0.0,0.0,1.0,0.0,0.0,0.0625,0.0625,0.1875,0.0625,0.1250], [2, "Text to model with",0.0,0.0,0.0,0.0,0.0,0.1333,0.1333,0.0667,0.0667,0.0667], [3, "Text to work with",0.0,0.0,0.0,0.0,0.0,0.1111,0.0938,0.0393,0.0198,0.2759], [4, "Free text comments",0.0,0.0,1.0,1.0,1.0,0.2162,0.1104,0.0341,0.0847,0.0559] ]df = pd.DataFrame(input_data, columns = column_headers)df = df.set_index('id')df我觉得我的实现有点迂回，因为 FeatureUnion 只能处理合并时的二维数组，所以任何其他类型如 DataFrame 对我来说都是有问题的。然而，这个例子是可行的——我只是在寻找改进它的方法，使其更加简洁（DRY）。
from sklearn.pipeline import Pipeline, FeatureUnionfrom sklearn.base import BaseEstimator, TransformerMixinclass ItemSelector(BaseEstimator, TransformerMixin): def __init__(self, column): self.column = column def fit(self, X, y=None): return self def transform(self, X, y=None): return X[self.column]class ArrayCaster(BaseEstimator, TransformerMixin): def fit(self, x, y=None): return self def transform(self, data): return np.transpose(np.matrix(data))def basic_text_model(trainX, testX, trainY, testY, classLabels, plotPath): '''OneVsRestClassifier for multi-label prediction''' pipeline = Pipeline([ ('features', FeatureUnion([ ('topic1_feature', Pipeline([ ('selector', ItemSelector(column='t1_prob')), ('caster', ArrayCaster()) ])), ('topic2_feature', Pipeline([ ('selector', ItemSelector(column='t2_prob')), ('caster', ArrayCaster()) ])), ('topic3_feature', Pipeline([ ('selector', ItemSelector(column='t3_prob')), ('caster', ArrayCaster()) ])), ('topic4_feature', Pipeline([ ('selector', ItemSelector(column='t4_prob')), ('caster', ArrayCaster()) ])), ('topic5_feature', Pipeline([ ('selector', ItemSelector(column='t5_prob')), ('caster', ArrayCaster()) ])), ('word_features', Pipeline([ ('vect', CountVectorizer(analyzer="word", stop_words='english')), ('tfidf', TfidfTransformer(use_idf = True)), ])), ])), ('clf', OneVsRestClassifier(svm.LinearSVC(random_state=random_state))) ])# Fit the modelpipeline.fit(trainX, trainY)predicted = pipeline.predict(testX)我将 ArrayCaster 纳入流程的想法来自于这个 回答。
回答：
我通过使用 FunctionTransformer 解决了这个问题，灵感来自于 @Marcus V 对这个 问题 的解决方案。修订后的管道更加简洁。
from sklearn.preprocessing import FunctionTransformerget_numeric_data = FunctionTransformer(lambda x: x[['t1_prob', 't2_prob', 't3_prob', 't4_prob', 't5_prob']], validate=False)pipeline = Pipeline( [ ( "features", FeatureUnion( [ ("numeric_features", Pipeline([("selector", get_numeric_data)])), ( "word_features", Pipeline( [ ("vect", CountVectorizer(analyzer="word", stop_words="english")), ("tfidf", TfidfTransformer(use_idf=True)), ] ), ), ] ), ), ("clf", OneVsRestClassifier(svm.LinearSVC(random_state=10))), ])