我有一个包含284个特征的数据集,试图使用scikit-learn进行插补,然而我得到了一个错误,特征数量变为了283个:
imputer = SimpleImputer(missing_values = np.nan, strategy = "mean")imputer = imputer.fit(data.iloc[:,0:284])df[:,0:284] = imputer.transform(df[:,0:284])X = MinMaxScaler().fit_transform(df)---------------------------------------------------------------------------ValueError Traceback (most recent call last)<ipython-input-150-849be5be8fcb> in <module> 1 imputer = SimpleImputer(missing_values = np.nan, strategy = "mean") 2 imputer = imputer.fit(data.iloc[:,0:284])----> 3 df[:,0:284] = imputer.transform(df[:,0:284]) 4 X = MinMaxScaler().fit_transform(df)~\Anaconda3\envs\environment\lib\site-packages\sklearn\impute\_base.py in transform(self, X) 411 if X.shape[1] != statistics.shape[0]: 412 raise ValueError("X has %d features per sample, expected %d"--> 413 % (X.shape[1], self.statistics_.shape[0])) 414 415 # Delete the invalid columns if strategy is not constantValueError: X has 283 features per sample, expected 284
我不明白为什么会变成283个特征,我猜测在拟合过程中它可能找到了所有值为0的特征并决定删除这些特征,但我找不到任何文档说明如何确保这些特征被保留。我不是程序员,所以不确定是否遗漏了什么显而易见的东西,或者是否应该考虑其他方法?
回答:
如果你的数据集中有某个特征完全没有值,就会发生这种情况,根据https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html的说明:“如果策略不是‘constant’,在fit时仅包含缺失值的列在transform时会被丢弃”。你可以通过在构建imputer时使用较高的’verbose’值来检查是否确实是这个问题:
sklearn.impute.SimpleImputer(…, verbose=100,…)
它会输出类似这样的警告:UserWarning: Deleting features without observed values: [ … ]