我目前正在尝试根据这篇论文编写注意力机制：“Effective Approaches to Attention-based Neural Machine Translation”, Luong, Pham, Manning (2015)。（我使用的是带有点积分数的全局注意力）。

然而，我不确定如何将LSTM解码器的隐藏状态和输出状态输入进去。问题在于，在时间t时，LSTM解码器的输入依赖于我需要使用t-1时刻的输出和隐藏状态计算的量。

这是代码的相关部分：

with tf.variable_scope('data'):    prob = tf.placeholder_with_default(1.0, shape=())    X_or = tf.placeholder(shape = [batch_size, timesteps_1, num_input], dtype = tf.float32, name = "input")    X = tf.unstack(X_or, timesteps_1, 1)    y = tf.placeholder(shape = [window_size,1], dtype = tf.float32, name = "label_annotation")    logits = tf.zeros((1,1), tf.float32)with tf.variable_scope('lstm_cell_encoder'):    rnn_layers = [tf.nn.rnn_cell.LSTMCell(size) for size in [hidden_size, hidden_size]]    multi_rnn_cell = tf.nn.rnn_cell.MultiRNNCell(rnn_layers)    lstm_outputs, lstm_state =  tf.contrib.rnn.static_rnn(cell=multi_rnn_cell,inputs=X,dtype=tf.float32)    concat_lstm_outputs = tf.stack(tf.squeeze(lstm_outputs))    last_encoder_state = lstm_state[-1]with tf.variable_scope('lstm_cell_decoder'):    initial_input = tf.unstack(tf.zeros(shape=(1,1,hidden_size2)))    rnn_decoder_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, state_is_tuple = True)    # 计算h_1的隐藏和输出    for index in range(window_size):        output_decoder, state_decoder = tf.nn.static_rnn(rnn_decoder_cell, initial_input, initial_state=last_encoder_state, dtype=tf.float32)        # 计算源输出向量的分数        scores = tf.matmul(concat_lstm_outputs, tf.reshape(output_decoder[-1],(hidden_size,1)))        attention_coef = tf.nn.softmax(scores)        context_vector = tf.reduce_sum(tf.multiply(concat_lstm_outputs, tf.reshape(attention_coef, (window_size, 1))),0)        context_vector = tf.reshape(context_vector, (1,hidden_size))        # 计算tilda隐藏状态 \tilde{h}_t=tanh(W[c_t, h_t]+b_t)        concat_context = tf.concat([context_vector, output_decoder[-1]], axis = 1)        W_tilde = tf.Variable(tf.random_normal(shape = [hidden_size*2, hidden_size2], stddev = 0.1), name = "weights_tilde", trainable = True)        b_tilde = tf.Variable(tf.zeros([1, hidden_size2]), name="bias_tilde", trainable = True)        hidden_tilde = tf.nn.tanh(tf.matmul(concat_context, W_tilde)+b_tilde) # hidden_tilde是[1*64]        # 更新到下一个时间步        initial_input = tf.unstack(tf.reshape(hidden_tilde, (1,1,hidden_size2)))        last_encoder_state = state_decoder        # 预测目标        W_target = tf.Variable(tf.random_normal(shape = [hidden_size2, 1], stddev = 0.1), name = "weights_target", trainable = True)        logit = tf.matmul(hidden_tilde, W_target)        logits = tf.concat([logits, logit], axis = 0)    logits = logits[1:]

循环内的部分是我不确定的。当我覆盖变量”initial_input”和”last_encoder_state”时，TensorFlow是否会记住计算图？

回答：

我认为如果你使用tf.contrib.seq2seq.AttentionWrapper及其实现之一：BahdanauAttention或LuongAttention，你的模型将会简化很多。

这样，你就可以在单元级别上连接注意力向量，使得单元输出在应用注意力之后就已经是最终结果。来自seq2seq教程的示例：

cell = LSTMCell(512)attention_mechanism = tf.contrib.seq2seq.LuongAttention(512, encoder_outputs)attn_cell = tf.contrib.seq2seq.AttentionWrapper(cell, attention_mechanism, attention_size=256)

请注意，这样你就不需要window_size的循环了，因为tf.nn.static_rnn或tf.nn.dynamic_rnn会实例化带有注意力的单元。

关于你的问题：你应该区分Python变量和TensorFlow图节点：你可以将last_encoder_state赋值给不同的张量，原始图节点不会因为这个操作而改变。这很灵活，但也可能在结果网络中造成误导 – 你可能认为你将LSTM连接到一个张量，但实际上是另一个。一般来说，你不应该这样做。