我正在尝试将我的文本分类模型进行 pickle 处理,并重新加载到 Flask 应用程序界面中。
我有一个特定的函数作为分析器使用,称为 split_into_lemmas
def split_into_lemmas(message): message = unicode(message, 'utf8').lower() words = TextBlob(message).words # 对每个单词,取其“基础形式” = 词形还原 return [word.lemma for word in words]from sklearn.pipeline import Pipelinecount_vect = CountVectorizer(analyzer=split_into_lemmas,ngram_range= (1, 3), encoding='utf8',stop_words =None)tfidf_transformer = TfidfTransformer()text_clf = Pipeline([('vect', count_vect), ('tdif', tfidf_transformer), ('clf', best_svc)])%%timetext_clf.fit(X=data['Condition'], y=data['condition_predict'])
我训练了模型并通过 pickle 保存它
_ = joblib.dump(text_clf, 'classification_pipeline.pkl')
另一方面
当我尝试重新加载管道时
我得到了以下错误
---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<ipython-input-3-bb0859b3946a> in <module>() 6 7 clf_pipeline = open('C:/Users/Falco/Desktop/directory/WRMD_paper/classification_pipeline.pkl','rb')----> 8 clf = joblib.load(clf_pipeline)C:\ProgramData\Anaconda2\lib\site-packages\sklearn\externals\joblib\numpy_pickle.pyc in load(filename, mmap_mode) 586 filename = getattr(fobj, 'name', '') 587 with _read_fileobject(fobj, filename, mmap_mode) as fobj:--> 588 obj = _unpickle(fobj) 589 else: 590 with open(filename, 'rb') as f:C:\ProgramData\Anaconda2\lib\site-packages\sklearn\externals\joblib\numpy_pickle.pyc in _unpickle(fobj, filename, mmap_mode) 524 obj = None 525 try:--> 526 obj = unpickler.load() 527 if unpickler.compat_mode: 528 warnings.warn("The file '%s' has been generated with a "C:\ProgramData\Anaconda2\lib\pickle.pyc in load(self) 862 while 1: 863 key = read(1)--> 864 dispatch[key](self) 865 except _Stop, stopinst: 866 return stopinst.valueC:\ProgramData\Anaconda2\lib\pickle.pyc in load_global(self) 1094 module = self.readline()[:-1] 1095 name = self.readline()[:-1]-> 1096 klass = self.find_class(module, name) 1097 self.append(klass) 1098 dispatch[GLOBAL] = load_globalC:\ProgramData\Anaconda2\lib\pickle.pyc in find_class(self, module, name) 1130 __import__(module) 1131 mod = sys.modules[module]-> 1132 klass = getattr(mod, name) 1133 return klass 1134 AttributeError: 'module' object has no attribute 'split_into_lemmas'
当我在笔记本中重新声明该函数时,模型加载良好并运行,但当我将笔记本保存为 .py 文件并作为 Flask 应用程序运行时,它不运行并给我相同的错误。
有人能帮助我正确保存管道,这样我就不必声明该函数了吗?
回答:
当你重新加载 pickle 时,你也需要定义 split_into_lemmas 函数。