我刚开始学习机器学习,正在使用 Python(3.6)、Pandas、Numpy 和 SKLearn 进行一个项目。
我的 DataFrame 是这样的:
discount tax total subtotal productid
3 0 20 13 002
10 3 106 94 003
46.49 6 21 20 004
我进行分类的方法如下:
df_full = pd.read_excel('input/Potential_Learning_Patterns.xlsx', sheet_name=0)
df_full.head()
# 转换为数值型
df_full['discount'] = pd.to_numeric(df_full['discount'], errors='coerce')
df_full['productdiscount'] = pd.to_numeric(df_full['discount'], errors='coerce')
df_full['Class'] = ((df_full['discount'] > 20) &
(df_full['tax'] == 0) &
(df_full['productdiscount'] > 20) &
(df_full['total'] > 100)).astype(int)
print (df_full)
# 从整个数据集中获取一些样本数据
data = df_full.sample(frac = 0.1, random_state = 1)
print(data.shape)
data.isnull().sum()
# 将 Excel 数据转换为矩阵
columns = "invoiceid locationid timestamp customerid discount tax total subtotal productid quantity productprice productdiscount invoice_products_id producttax invoice_payments_id paymentmethod paymentdetails amount Class(0/1) Class".split()
X = pd.DataFrame.as_matrix(data, columns=columns)
Y = data.Class
# temp = np.array(temp).reshape((len(temp), 1)
Y = Y.values.reshape(Y.shape[0], 1)
X.shape
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.06)
X_test, X_dev, Y_test, Y_dev = train_test_split(X_test, Y_test, test_size = .5)
# 检查训练集和其他集合中是否有分类值 - 0/1
np.where(Y_train == 1)
np.where(Y_test == 1)
np.where(Y_dev == 1)
# 确定数据集中欺诈案例的数量
Fraud = data[data['Class'] == 1]
Valid = data[data['Class'] == 0]
# 计算欺诈和有效案例的百分比
outlier_fraction = len(Fraud) / float(len(Valid))
print(outlier_fraction)
print('Fraud Cases : {}'.format(len(Fraud)))
print('Valid Cases : {}'.format(len(Valid)))
# 相关矩阵
corrmat = data.corr()
fig = plt.figure( figsize = (12, 9))
sns.heatmap(corrmat, vmax = .8, square = True)
plt.show()
我应用reshape的方法如下:
# 获取数据框中的所有列
columns = data.columns.tolist()
# 过滤不需要的数据列
columns = [c for c in columns if c not in ["Class"] ]
# 存储我们要预测的变量
target = "Class"
for column in data.columns:
if data[column].dtype == type(object):
le = LabelEncoder()
data[column] = le.fit_transform(data[column])
X = data[column]
X = data[column]
Y = data[target]
# 打印 X 和 Y 的形状
print(X.shape)
print(Y.shape)
# 定义一个随机状态
state = 1
# 定义异常检测方法
classifiers = {
"Isolation Forest": IsolationForest(max_samples=len(X),
contamination=outlier_fraction,
random_state=state),
"Local Outlier Factor": LocalOutlierFactor(
n_neighbors = 20,
contamination = outlier_fraction)}
# 拟合模型
n_outliers = len(Fraud)
for i, (clf_name, clf) in enumerate(classifiers.items()):
# 拟合数据并标记异常值
if clf_name == "Local Outlier Factor":
y_pred = clf.fit_predict(X)
scores_pred = clf.negative_outlier_factor_
else:
clf.fit(X)
scores_pred = clf.decision_function(X)
y_pred = clf.predict(X)
# 将预测值重塑为0表示有效,1表示欺诈
y_pred[y_pred == 1] = 0
y_pred[y_pred == -1] = 1
n_errors = (y_pred != Y).sum()
# 运行分类指标
print('{}:{}'.format(clf_name, n_errors))
print(accuracy_score(Y, y_pred ))
print(classification_report(Y, y_pred ))
代码在重塑样本和目标之前运行正常。但是当我尝试对分类器使用fit方法时,它返回了一个错误,如下所示:
ValueError: Expected 2D array, got 1D array instead: array=[1 0]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
我刚开始学习机器学习,我在这里做错了什么?我有多个特征,我如何正确地重塑我的样本数组?
请帮助我!提前感谢!
回答:
在下面的循环中,你在每次循环迭代中用单个列(Series)覆盖了变量X
:
for column in data.columns:
if data[column].dtype == type(object):
le = LabelEncoder()
data[column] = le.fit_transform(data[column])
X = data[column]
X = data[column]
# <------- 注意:
Y = data[target]
实际上,你可以在循环之后定义X
和Y
如下:
X = data.drop(target, 1)
Y = data[target]
绝大多数sklearn
方法都接受 pandas DataFrames 和 Series 作为输入数据集…