当我手动计算二元交叉熵时，我会先对logits应用sigmoid函数得到概率，然后使用交叉熵公式并计算结果的平均值：
logits = tf.constant([-1, -1, 0, 1, 2.])labels = tf.constant([0, 0, 1, 1, 1.])probs = tf.nn.sigmoid(logits)loss = labels * (-tf.math.log(probs)) + (1 - labels) * (-tf.math.log(1 - probs))print(tf.reduce_mean(loss).numpy()) # 0.35197204cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)loss = cross_entropy(labels, logits)print(loss.numpy()) # 0.35197204当logits和labels大小不同时，如何计算分类交叉熵？
logits = tf.constant([[-3.27133679, -22.6687183, -4.15501118, -5.14916372, -5.94609261, -6.93373299, -5.72364092, -9.75725174, -3.15748906, -4.84012318], [-11.7642536, -45.3370094, -3.17252636, 4.34527206, -17.7164974, -0.595088899, -17.6322937, -2.36941719, -6.82157373, -3.47369862], [-4.55468369, -1.07379043, -3.73261762, -7.08982277, -0.0288562477, -5.46847963, -0.979336262, -3.03667569, -3.29502845, -2.25880361]])labels = tf.constant([2, 3, 4])loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')loss = loss_object(labels, logits)print(loss.numpy()) # [2.0077195 0.00928135 0.6800677 ]print(tf.reduce_mean(loss).numpy()) # 0.8990229我的意思是如何手动计算得到相同的结果（[2.0077195 0.00928135 0.6800677 ]）？
@[隐藏人名]的回答是正确的。在TF 2.0中看起来是这样的：
loss_object = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True, reduction='none')loss = loss_object(labels, logits)print(f'{loss.numpy()}\n{tf.math.reduce_sum(loss).numpy()}')one_hot_labels = tf.one_hot(labels, 10)preds = tf.nn.softmax(logits)preds /= tf.math.reduce_sum(preds, axis=-1, keepdims=True)loss = tf.math.reduce_sum(tf.math.multiply(one_hot_labels, -tf.math.log(preds)), axis=-1)print(f'{loss.numpy()}\n{tf.math.reduce_sum(loss).numpy()}')# [2.0077195 0.00928135 0.6800677 ]# 2.697068691253662# [2.0077198 0.00928142 0.6800677 ]# 2.697068929672241对于语言模型：
vocab_size = 9seq_len = 6batch_size = 2labels = tf.reshape(tf.range(batch_size*seq_len), (batch_size,seq_len)) # (2, 6)logits = tf.random.normal((batch_size,seq_len,vocab_size)) # (2, 6, 9)loss_object = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True, reduction='none')loss = loss_object(labels, logits)print(f'{loss.numpy()}\n{tf.math.reduce_sum(loss).numpy()}')one_hot_labels = tf.one_hot(labels, vocab_size)preds = tf.nn.softmax(logits)preds /= tf.math.reduce_sum(preds, axis=-1, keepdims=True)loss = tf.math.reduce_sum(tf.math.multiply(one_hot_labels, -tf.math.log(preds)), axis=-1)print(f'{loss.numpy()}\n{tf.math.reduce_sum(loss).numpy()}')# [[1.341706 3.2518263 2.6482694 3.039099 1.5835983 4.3498387]# [2.67237 3.3978183 2.8657475 nan nan nan]]# nan# [[1.341706 3.2518263 2.6482694 3.039099 1.5835984 4.3498387]# [2.67237 3.3978183 2.8657475 0. 0. 0. ]]# 25.1502742767334回答：
SparseCategoricalCrossentropy是CategoricalCrossentropy的一种，它接受的是整数标签而不是独热编码。来自源代码的示例，下面两个是等价的：
scce = tf.keras.losses.SparseCategoricalCrossentropy()cce = tf.keras.losses.CategoricalCrossentropy()labels_scce = K.variable([[0, 1, 2]]) labels_cce = K.variable([[1, 0, 0], [0, 1, 0], [0, 0, 1]])preds = K.variable([[.90,.05,.05], [.50,.89,.60], [.05,.01,.94]])loss_cce = cce(labels_cce, preds, from_logits=False)loss_scce = scce(labels_scce, preds, from_logits=False)with tf.Session() as sess: sess.run(tf.global_variables_initializer()) sess.run([loss_cce, loss_scce])print(K.get_value(loss_cce))print(K.get_value(loss_scce))# [0.10536055 0.8046684 0.0618754]# [0.10536055 0.8046684 0.0618754]至于如何手动计算，我们可以参考Numpy后端：
np_labels = K.get_value(labels_cce)np_preds = K.get_value(preds)losses = []for label, pred in zip(np_labels, np_preds): pred /= pred.sum(axis=-1, keepdims=True) losses.append(np.sum(label * -np.log(pred), axis=-1, keepdims=False))print(losses)# [0.10536055 0.8046684 0.0618754]from_logits = True：preds是模型输出在进入softmax之前（所以我们需要将其传递给softmax）from_logits = False：preds是模型输出在经过softmax之后（所以我们跳过这一步）
总结一下，手动计算的步骤如下：
- 将整数标签转换为独热编码标签
- 如果
preds是softmax之前的模型输出，我们需要计算它们的softmax pred /= ...在计算对数之前归一化预测值；这样，高概率的预测在零标签上会惩罚在非零标签上的正确预测。如果from_logits = False，这一步会被跳过，因为softmax已经完成了归一化。参见这个片段。进一步阅读- 对于每个观测值/样本，计算元素级的负
log（以e为底），仅在label==1的地方 - 计算所有观测值的损失的平均值
最后，分类交叉熵的数学公式是：
i遍历N个观测值c遍历C个类别- 1是指示函数 – 这里类似于二元交叉熵，但操作的是长度为
C的向量 p_model [y_i \in C_c]– 观测值i属于类别c的预测概率
