我有一段使用TensorFlow 1的Keras代码。代码修改了损失函数以进行深度强化学习:
import osimport gymimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltenv = gym.make("CartPole-v0").envenv.reset()n_actions = env.action_space.nstate_dim = env.observation_space.shapefrom tensorflow import kerasimport randomfrom tensorflow.keras import layers as Limport tensorflow as tffrom tensorflow.python.keras.backend import set_sessionsess = tf.compat.v1.Session()graph = tf.compat.v1.get_default_graph()init = tf.global_variables_initializer()sess.run(init)network = keras.models.Sequential()network.add(L.InputLayer(state_dim))# 让我们创建一个遵循上述指南的近似Q学习的网络network.add(L.Dense(5, activation='elu'))network.add(L.Dense(5, activation='relu'))network.add(L.Dense(n_actions, activation='linear'))s = env.reset()# 为<s, a, r, s'>元组创建占位符，并为游戏结束创建一个特殊指示符（is_done = True）states_ph = keras.backend.placeholder(dtype='float32', shape=(None,) + state_dim)actions_ph = keras.backend.placeholder(dtype='int32', shape=[None])rewards_ph = keras.backend.placeholder(dtype='float32', shape=[None])next_states_ph = keras.backend.placeholder(dtype='float32', shape=(None,) + state_dim)is_done_ph = keras.backend.placeholder(dtype='bool', shape=[None])#获取当前状态下所有动作的Q值predicted_qvalues = network(states_ph)#选择所选动作的Q值predicted_qvalues_for_actions = tf.reduce_sum(predicted_qvalues * tf.one_hot(actions_ph, n_actions), axis=1)gamma = 0.99# 计算下一状态下所有动作的Q值predicted_next_qvalues = network(next_states_ph)# 使用预测的下一Q值计算V*(next_states)next_state_values = tf.math.reduce_max(predicted_next_qvalues, axis=1)# 计算用于损失的“目标Q值” - 这是上述公式中方括号内的内容target_qvalues_for_actions = rewards_ph + tf.constant(gamma) * next_state_values# 在最后一个状态我们应该使用简化的公式: Q(s,a) = r(s,a) 因为s'不存在target_qvalues_for_actions = tf.where(is_done_ph, rewards_ph, target_qvalues_for_actions)#最小化的均方误差损失loss = (predicted_qvalues_for_actions - tf.stop_gradient(target_qvalues_for_actions)) ** 2loss = tf.reduce_mean(loss)# 类似于表格代理中agent.update(state, action, reward, next_state)的训练函数train_step = tf.compat.v1.train.AdamOptimizer(1e-4).minimize(loss)a = 0next_s, r, done, _ = env.step(a)sess.run(train_step, { states_ph: [s], actions_ph: [a], rewards_ph: [r], next_states_ph: [next_s], is_done_ph: [done] })当我运行sess.run()训练步骤时，我得到了以下错误:
tensorflow.python.framework.errors_impl.FailedPreconditionError: 从容器localhost读取资源变量beta1_power时出错。这可能意味着变量未初始化。未找到: 容器localhost不存在。（找不到资源: localhost/beta1_power）有什么想法可能是什么问题吗?
回答：
初始化操作应该在变量（即模型）创建或计算图定义之后被获取并运行（仅一次）。因此，它们应该放在运行训练步骤之前:
# 定义并创建计算图/模型# ...# 初始化图/模型中的变量init = tf.global_variables_initializer()sess.run(init)# 开始训练sess.run(train_step, ...)