Keras保存的模型在不同会话中预测不同值

我训练了一个命名实体识别模型,保存后再加载,在同一个IPython会话中可以正确预测,但每当我关闭会话并重新打开时,加载的模型预测结果会变得随机。你能帮我解决这个问题吗?

我使用以下方式将模型保存为hdf5格式:

Model.save("filename")

并使用以下方式加载模型:

Model.load_model("filename")

这是我的完整代码

import pandas as pdimport numpy as npimport osfrom keras.preprocessing.sequence import pad_sequencesfrom keras.utils import to_categoricalfrom sklearn.model_selection import train_test_splitfrom keras.models import Model, Input,load_modelfrom keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout,  Bidirectionalfrom nltk import pos_tag, word_tokenize,sent_tokenizedata = pd.read_csv("E:\ml tut\entity recognition\exdataset.csv", encoding="latin1")data = data.fillna(method="ffill")words = list(set(data["Word"].values))words.append("ENDPAD")n_words = len(words); n_wordstags = list(set(data["Tag"].values))n_tags = len(tags); n_tagsclass SentenceGetter(object):    def __init__(self, data):        self.n_sent = 1        self.data = data        self.empty = False        agg_func = lambda s: [((w, p), t) for w, p, t in                  zip(s["Word"].values.tolist(),s["POS"].values.tolist(),                                                  s["Tag"].values.tolist())]        self.grouped = self.data.groupby("Sentence #").apply(agg_func)        self.sentences = [s for s in self.grouped]    def get_next(self):        try:            s = self.grouped["Sentence: {}".format(self.n_sent)]            self.n_sent += 1            return s        except:            return Nonegetter = SentenceGetter(data)sent = getter.get_next()print(sent)sentences = getter.sentencesmax_len = 50word2idx = {w: i for i, w in enumerate(words)}tag2idx = {t: i for i, t in enumerate(tags)}input = Input(shape=(max_len,))model = Embedding(input_dim=n_words, output_dim=50, input_length=max_len)        (input)model = Dropout(0.1)(model)model = Bidirectional(LSTM(units=100, return_sequences=True, recurrent_dropout=0.1))(model)out = TimeDistributed(Dense(n_tags, activation="softmax"))(model)  if os.path.exists('my_model.h5'):    print("loading model")    model = load_model('my_model.h5')else:    print("training model")    X = [[word2idx[w[0][0]] for w in s] for s in sentences]    X = pad_sequences(maxlen=max_len, sequences=X, padding="post",     value=n_words - 1)    y = [[tag2idx[w[1]] for w in s] for s in sentences]    y = pad_sequences(maxlen=max_len, sequences=y, padding="post",     value=tag2idx["O"])    y = [to_categorical(i, num_classes=n_tags) for i in y]    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.1)    model = Model(input, out)    model.compile(optimizer="rmsprop", loss="categorical_crossentropy",     metrics=["accuracy"])    model.fit(X_tr, np.array(y_tr), batch_size=32, epochs=5,     validation_split=0.1, verbose=1)    model.save('my_model.h5')    my_input="Albert Einstein is a great guy,he lives in berlin, Germany."    print("--------------")     test_sentence = word_tokenize(my_input)    x_test_sent = pad_sequences(sequences=[[word2idx.get(w, 0) for w in     test_sentence]],padding="post", value=0, maxlen=max_len)    i = 0    p = model.predict(np.array([x_test_sent[i]]))    p = np.argmax(p, axis=-1)    print("{:15}||{}".format("Word", "Prediction"))    print(30 * "=")    for w, pred in zip(test_sentence, p[0]):    if w != 0:         print("{:15}: {}".format(w, tags[pred]))

回答:

请在生成模型时将你的标签(tags = list(set(data[“Tag”].values)))保存为pickle文件。这将解决你的问题。

因此,你需要保存以下内容:1. 标签 2. 模型 3. word2idx

Related Posts

L1-L2正则化的不同系数

我想对网络的权重同时应用L1和L2正则化。然而,我找不…

使用scikit-learn的无监督方法将列表分类成不同组别,有没有办法?

我有一系列实例,每个实例都有一份列表,代表它所遵循的不…

f1_score metric in lightgbm

我想使用自定义指标f1_score来训练一个lgb模型…

通过相关系数矩阵进行特征选择

我在测试不同的算法时,如逻辑回归、高斯朴素贝叶斯、随机…

可以将机器学习库用于流式输入和输出吗?

已关闭。此问题需要更加聚焦。目前不接受回答。 想要改进…

在TensorFlow中,queue.dequeue_up_to()方法的用途是什么?

我对这个方法感到非常困惑,特别是当我发现这个令人费解的…

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注