我正在尝试创建一个Keras LSTM模型，用于将单词分类为0或1。然而，无论我输入什么文本，网络总是返回接近零的值。我已经将问题缩小到与Keras分词器相关的问题。我已经添加了一个调试打印语句，并注释了model.predict()代码来测试这个问题。所有单词都返回数组[[208]]。
以下是代码
from builtins import lenfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing.sequence import pad_sequencesfrom keras.models import Sequentialfrom keras import layersfrom sklearn.model_selection import train_test_splitimport pandas as pdimport numpy as npimport enchantimport red = enchant.Dict("en_US")df = pd.read_csv('sentiments.csv')df.columns = ["label", "text"]x = df['text'].valuesy = df['label'].valuesx_train, x_test, y_train, y_test = \ train_test_split(x, y, test_size=0.1, random_state=123)tokenizer = Tokenizer(num_words=100)tokenizer.fit_on_texts(x)xtrain = tokenizer.texts_to_sequences(x_train)xtest = tokenizer.texts_to_sequences(x_test)vocab_size = len(tokenizer.word_index) + 1maxlen = 10xtrain = pad_sequences(xtrain, padding='post', maxlen=maxlen)xtest = pad_sequences(xtest, padding='post', maxlen=maxlen)print(x_train[3])print(xtrain[3])embedding_dim = 50model = Sequential()model.add(layers.Embedding(input_dim=(vocab_size+1), output_dim=embedding_dim, input_length=maxlen))model.add(layers.LSTM(units=50, return_sequences=True))model.add(layers.LSTM(units=10))model.add(layers.Dropout(0.5))model.add(layers.Dense(8))model.add(layers.Dense(1, activation="sigmoid"))model.compile(optimizer="adam", loss="binary_crossentropy", metrics=['accuracy'])model.summary()model.fit(xtrain, y_train, epochs=20, batch_size=16, verbose=False)loss, acc = model.evaluate(xtrain, y_train, verbose=False)print("Training Accuracy: ", acc)loss, acc = model.evaluate(xtest, y_test, verbose=False)print("Test Accuracy: ", acc)text_input = str(input("Enter a word for analysis: "))if d.check(text_input): word_Arr = [] word_Arr.append(text_input) tokenizer.fit_on_texts(word_Arr) word_final = tokenizer.texts_to_sequences(word_Arr) word_final_final = np.asarray(word_final) print(word_final_final) # newArr = np.zeros(shape=(6, 10)) # newArr[0] = word_final_final # print(model.predict(newArr))我该如何继续？
回答：
您总是重新拟合您的Tokenizer实例：
tokenizer = Tokenizer(num_words=100)tokenizer.fit_on_texts(x)用新输入的单词本身：
tokenizer.fit_on_texts(word_Arr)因此，您创建的标记（您用来训练模型的标记）将被删除，而您重新拟合的Token实例将根据您输入的单词进行标记化。
示例：
tokenizer = Tokenizer(num_words=100)tokenizer.fit_on_texts(["dog, cat, horse"])ext_input = str(input("Enter a word for analysis: "))word_Arr = []word_Arr.append(text_input)# 这里是您的问题！！！tokenizer.fit_on_texts(word_Arr)word_final = tokenizer.texts_to_sequences(word_Arr)word_final_final = np.asarray(word_final)print(word_final_final)输出：
Enter a word for analysis: dog[[1]]Enter a word for analysis: cat[[1]]注释掉有问题的代码部分：
tokenizer = Tokenizer(num_words=100)tokenizer.fit_on_texts(["dog, cat, horse"])ext_input = str(input("Enter a word for analysis: "))word_Arr = []word_Arr.append(text_input)# 注释掉您的问题！！！# tokenizer.fit_on_texts(word_Arr)word_final = tokenizer.texts_to_sequences(word_Arr)word_final_final = np.asarray(word_final)print(word_final_final)输出
Enter a word for analysis: cat[[2]]Enter a word for analysis: dog[[1]]