我希望创建一个通用函数，将sklearn的IsolationForest的输出decision_scores转换为真正的概率[0.0, 1.0]。
我已经阅读并了解了原始论文，我明白从数学角度来看，该函数的输出不是概率，而是每个基础估计器构建的路径长度的平均值，用于隔离异常值。
问题
我想将该输出转换为概率形式的tuple (x,y)，其中x=P(anomaly)和y=1-x。
当前方法
def convert_probabilities(predictions, scores): from sklearn.preprocessing import MinMaxScaler new_scores = [(1,1) for _ in range(len(scores))] anomalous_idxs = [i for i in (range(len(predictions))) if predictions[i] == -1] regular_idxs = [i for i in (range(len(predictions))) if predictions[i] == 1] anomalous_scores = np.asarray(np.abs([scores[i] for i in anomalous_idxs])) regular_scores = np.asarray(np.abs([scores[i] for i in regular_idxs])) scaler = MinMaxScaler() anomalous_scores_scaled = scaler.fit_transform(anomalous_scores.reshape(-1,1)) regular_scores_scaled = scaler.fit_transform(regular_scores.reshape(-1,1)) for i, j in zip(anomalous_idxs, range(len(anomalous_scores_scaled))): new_scores[i] = (anomalous_scores_scaled[j][0], 1-anomalous_scores_scaled[j][0]) for i, j in zip(regular_idxs, range(len(regular_scores_scaled))): new_scores[i] = (1-regular_scores_scaled[j][0], regular_scores_scaled[j][0]) return new_scoresmodified_scores = convert_probabilities(model_predictions, model_decisions)最小可重现示例
import pandas as pdfrom sklearn.datasets import make_classification, load_irisfrom sklearn.ensemble import IsolationForestfrom sklearn.decomposition import PCAfrom sklearn.model_selection import train_test_split# Get dataX, y = load_iris(return_X_y=True, as_frame=True)anomalies, anomalies_classes = make_classification(n_samples=int(X.shape[0]*0.05), n_features=X.shape[1], hypercube=False, random_state=60, shuffle=True)anomalies_df = pd.DataFrame(data=anomalies, columns=X.columns)# Split into train/testX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=60)# Combine testing dataX_test['anomaly'] = 1anomalies_df['anomaly'] = -1X_test = X_test.append(anomalies_df, ignore_index=True)y_test = X_test['anomaly']X_test.drop('anomaly', inplace=True, axis=1)# Build a modelmodel = IsolationForest(n_jobs=1, bootstrap=False, random_state=60)# Fit itmodel.fit(X_train)# Test itmodel_predictions = model.predict(X_test)model_decisions = model.decision_function(X_test)# Print resultsfor a,b,c in zip(y_test, model_predictions, model_decisions): print_str = """ Class: {} | Model Prediction: {} | Model Decision Score: {} """.format(a,b,c) print(print_str)问题
modified_scores = convert_probabilities(model_predictions, model_decisions)# Print resultsfor a,b in zip(model_predictions, modified_scores): ans = False if a==-1: if b[0] > b[1]: ans = True else: ans = False elif a==1: if b[1] > b[0]: ans=True else: ans=False print_str = """ Model Prediction: {} | Model Decision Score: {} | Correct: {} """.format(a,b, str(ans)) print(print_str)显示了一些奇怪的结果，例如：
Model Prediction: 1 | Model Decision Score: (0.17604259932311161, 0.8239574006768884) | Correct: TrueModel Prediction: 1 | Model Decision Score: (0.7120367886017022, 0.28796321139829784) | Correct: FalseModel Prediction: 1 | Model Decision Score: (0.7251531538304419, 0.27484684616955807) | Correct: FalseModel Prediction: -1 | Model Decision Score: (0.16776449326185877, 0.8322355067381413) | Correct: FalseModel Prediction: 1 | Model Decision Score: (0.8395087028516501, 0.1604912971483499) | Correct: FalseModel Prediction: 1 | Model Decision Score: (0.0, 1.0) | Correct: True
如何可能预测为-1 (异常)，但概率仅为37%？或者预测为1 (正常)，但概率为26%？
请注意，玩具数据集是有标签的，但显然无监督的异常检测算法假设没有标签。
回答：
尽管几个月后，这个问题有了答案。
2011年发表了一篇论文，试图展示关于将异常分数统一为概率的研究。
事实上，pyod库有一个通用的predict_proba方法，该方法提供了使用这种统一方法的选项。
这是受他们的源代码启发的代码实现：
def convert_probabilities(data, model): decision_scores = model.decision_function(data) probs = np.zeros([data.shape[0], int(model.classes)]) pre_erf_score = ( decision_scores - np.mean(decision_scores) ) / ( np.std(decision_scores) * np.sqrt(2) ) erf_score = erf(pre_erf_score) probs[:, 1] = erf_score.clip(0, 1).ravel() probs[:, 0] = 1 - probs[:, 1] return probs（作为参考，pyod确实有一个Isolation Forest实现）
