我已经准备好了一个情感分析模型,并且试图用新的输入进行预测,但遇到了一个错误:
data = getFile('Cleaned Data.xlsx')data['Description'] = data['Description'].apply(lambda x: x.lower())data['Description'] = data['Description'].apply((lambda x: re.sub('[^a-zA-z0-9\s]','',x)))print(data[ data['Classification'] == 1].size)print(data[ data['Classification'] == 0].size)for idx,row in data.iterrows(): row[1] = row[1].replace('rt',' ')tokenizer = Tokenizer(split=' ')tokenizer.fit_on_texts(data['Description'].values)#vectorizer = CountVectorizer()#X = vectorizer.fit_transform(jobSpec['Description']).toarray()X = tokenizer.texts_to_sequences(data['Description'].values)X = pad_sequences(X, maxlen=100, value=0.)display(X.shape)vocab_size = len(tokenizer.word_index) + 1max_length = max([len(s.split()) for s in data['Description']])Y = pd.get_dummies(data['Classification']).valuesX_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.33, random_state = 42)tokens_docs = [doc.split(" ") for doc in data['Description'].values]all_tokens = itertools.chain.from_iterable(tokens_docs)my_dict = {token: token if token.isdigit() else idx for idx, token in enumerate(set(all_tokens))}print(X_train.shape,Y_train.shape)print(X_test.shape,Y_test.shape)raw_embedding = load_embedding('glove.6B.100d.txt')embedding_vectors= get_weight_matrix(raw_embedding, tokenizer.word_index)embedding_layer = Embedding(vocab_size, 100, weights=[embedding_vectors], input_length = X.shape[1], trainable=False, name="Embeddings")display(embedding_vectors.shape)# define modelsequence_input = Input(shape=(100,), dtype='int32')embedded_sequences = embedding_layer(sequence_input)x = Conv1D(128, 5, activation='relu')(embedded_sequences)x = MaxPooling1D(2)(x)x = Conv1D(128, 5, activation='relu')(x)x = MaxPooling1D(2)(x)x = Conv1D(128, 5, activation='relu')(x)x = MaxPooling1D(18)(x)x = Flatten()(x)x = Dense(128, activation='relu')(x)preds = Dense(2, activation='softmax')(x)model = Model(sequence_input, preds)model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])# input for which we need the embeddinginput_str = "the company our client is a renowned civil actor who have consistently and safely delivered major civil infrastructure ts across a"# build index based on our `vocabulary`word_to_idx = OrderedDict({w:all_tokens.index(w) for w in input_str.split() if w in all_tokens})ynew = model.predict([1],[3],[5],[7])display(ynew)
当我尝试用新的输入预测这个模型时:
ynew = model.predict([1],[3],[5],[7])display(ynew)
它给我返回了一个错误消息:
ValueError: Error when checking input: expected input_29 to have shape (100,) but got array with shape (1,)
我尝试更改模型的形状为None和1,但它会产生其他新的错误。我对机器学习方面还是比较新手,所以真的不确定如何解决这个问题。
任何帮助都将不胜感激
回答:
我通过更改输入的方式解决了这个问题:
text = ['Construction workers needed for this company who has qualifications ']#vectorizing the tweet by the pre-fitted tokenizer instancetext = tokenizer.texts_to_sequences(text)text = pad_sequences(twt, maxlen=100, dtype='int32', value=0)print(text)sentiment = model.predict(text)[0]display(sentiment)
我之前遇到错误是因为我没有将输入填充到与输入维度长度相匹配。