从Keras的IMDB数据集中恢复原始文本
我想从Keras的IMDB数据集中恢复IMDB的原始文本。
首先，当我加载Keras的IMDB数据集时，它返回的是词索引序列。
>>> (X_train, y_train), (X_test, y_test) = imdb.load_data()>>> X_train[0][1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 22665, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 21631, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 31050, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]我找到了imdb.get_word_index方法，它返回类似于{'create': 984, 'make': 94,…}的词索引字典。为了转换，我创建了索引词字典。
>>> word_index = imdb.get_word_index()>>> index_word = {v:k for k,v in word_index.items()}然后，我尝试像下面这样恢复原始文本。
>>> ' '.join(index_word.get(w) for w in X_train[5])"the effort still been that usually makes for of finished sucking ended cbc's an because before if just though something know novel female i i slowly lot of above freshened with connect in of script their that out end his deceptively i i"我的英语不好，但我知道这句话有些奇怪。
为什么会这样？我怎样才能恢复原始文本？
回答：
你的例子看起来像是乱码，远不止是缺少一些停用词那么简单。
如果你重新阅读start_char、oov_char和index_from参数的文档，它们解释了发生的情况，这些参数属于[keras.datasets.imdb.load_data](https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification)方法:
start_char: int. 序列的开始将用此字符标记。设置为1，因为0通常是填充字符。
oov_char: int. 由于num_words或skip_top限制而被删除的词将被此字符替换。
index_from: int. 使用此索引及更高的索引来索引实际的词。
你反转的那个字典假设词索引从1开始。
但是Keras返回的索引中，<START>和<UNKNOWN>分别是索引1和2。（并且它假设你会使用0作为<PADDING>）。
这对我有用：
import kerasNUM_WORDS=1000 # 只使用前1000个词INDEX_FROM=3 # 词索引偏移train,test = keras.datasets.imdb.load_data(num_words=NUM_WORDS, index_from=INDEX_FROM)train_x,train_y = traintest_x,test_y = testword_to_id = keras.datasets.imdb.get_word_index()word_to_id = {k:(v+INDEX_FROM) for k,v in word_to_id.items()}word_to_id["<PAD>"] = 0word_to_id["<START>"] = 1word_to_id["<UNK>"] = 2word_to_id["<UNUSED>"] = 3id_to_word = {value:key for key,value in word_to_id.items()}print(' '.join(id_to_word[id] for id in train_x[0] ))标点符号缺失，但仅此而已：
"<START> this film was just brilliant casting <UNK> <UNK> story direction <UNK> really <UNK> the part they played and you could just imagine being there robert <UNK> is an amazing actor ..."