我在尝试使用keras和IMDB数据集进行情感分析时遇到问题,当我尝试对文本进行标记化时,出现了一个键错误
import numpy as npfrom keras.datasets import imdbimport json(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=5000)from keras.preprocessing import sequenceX_train = sequence.pad_sequences(X_train, maxlen = 500)X_test = sequence.pad_sequences(X_test, maxlen = 500)from keras import Sequentialfrom keras.layers import Embedding, LSTM, Dense, Dropoutmodel = Sequential()model.add(Embedding(5000, 32, input_length = 500))model.add(LSTM(units = 100))model.add(Dense(1, activation = 'sigmoid'))print(model.summary())model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics=['accuracy'])model.fit(X_train, y_train, batch_size = 64, epochs = 6)good = "A great movie"bad = "This was not a great movie"from nltk import word_tokenizefrom keras.preprocessing import sequenceword2index = imdb.get_word_index()X=[]for word in word_tokenize(good): X.append(word2index[word])X=sequence.pad_sequences([X],maxlen=500)loaded_model.predict(X)
它没有进入预测部分,因为我得到了一个错误
File "<ipython-input-51-9268dcdfa83f>", line 9, in <module> test.append(word2index[word])KeyError: 'A'
我该如何解决这个问题?
回答:
避免使用大写单词并对字符串使用.lower()
是一个好习惯。
你的字符A
在word2index
字典中不存在,但a
存在。你会注意到word2index
中的每个元素都是小写的。
所以如果你使用X.append(word2index[word.lower()])
,你应该能得到正确的结果。