大家好，TensorFlow 的朋友们。

简而言之：

我在 Google Colab 上使用 TensorFlow 1.1.4 进行急切执行（eager execution）。

我的训练在第一个 epoch 的最后一个批次上失败了，因为我在训练循环中强制使用了批量大小的 tf.unpack，这有时不会得到一个完整的批次，并在以下情况失败：

Epoch 1/5147/148 [============================>.] - ETA: 0s - loss: 0.0133 - acc: 0.3863---------------------------------------------------------------------------InvalidArgumentError                      Traceback (most recent call last)<ipython-input-133-bba4aeb63f3b> in <module>()      1 steps_per_epoch=tf.ceil(len(all_image_paths)/BATCH_SIZE).numpy()      2 ----> 3 history = model.fit(ds, epochs=5, steps_per_epoch=steps_per_epoch)8 frames/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)InvalidArgumentError: Input shape axis 0 must equal 32, got shape [30]     [[{{node unstack}}]] [Op:IteratorGetNextSync]

我无法弄清楚如何解包一个包含 BATCH_SIZE 个文件路径（字符串张量）的张量，我将其解包、加载并预处理成 BATCH_SIZE 个图像张量：

def load_and_preprocess_image(batch_of_paths, batch_of_labels):  # 从我们的批次中解包我们的图像  list_of_paths = tf.unstack(batch_of_paths, num=BATCH_SIZE)  list_of_images = []  for path in list_of_paths:      image = tf.read_file(path)      if image == None:        print("无法加载路径上的图像：" + path )      image = preprocess_image(image)      list_of_images.append(image)  # 将我们现在加载并处理过的图像重新打包成一个批次  batch_of_images = tf.stack(list_of_images, 0)  return batch_of_images,batch_of_labels

有没有一种方法可以让我的 BATCH_SIZE 动态地从数据集传递的变量中读取，当一个批次不完整时？在我的情况下，我请求 BATCH_SIZE = 32，但在最后一次传递时得到 30 – 然后上述错误就出现了

更长的/完整的背景

我在 Google Colab 上使用 TensorFlow 1.1.4 进行急切执行（eager execution）。

我试图训练一个模型（多类别、多标签）来对 CSV 数据集进行训练，该数据集列出了图像文件路径，然后是列出标签是否存在、不存在或未知的列。

我还知道我的数据集包含很多未标记的数据（我们稍后会处理这个问题 :)）

标题和第一行看起来像这样：

filepath,color_key_blue,color_key_green,color_key_luma,color_key_matte,color_saturation_desaturated,color_saturation_neutral,color_saturation_pastel,color_saturation_saturated,color_theory_analagous,color_theory_complementary,color_theory_monochrome,color_tones_blackwhite,color_tones_cool,color_tones_warm,exterior,exterior_airplane,exterior_airport,exterior_apartment,exterior_auto_body,exterior_beach,exterior_bicycle,exterior_boat,exterior_bridge,exterior_bus,exterior_bus_stop,exterior_canyon,exterior_car,exterior_castle,exterior_cathedral,exterior_cave,exterior_church,exterior_city,exterior_desert,exterior_farm,exterior_forest,exterior_glacier,exterior_helicopter,exterior_hospital,exterior_house,exterior_industrial,exterior_lake,exterior_library,exterior_mall,exterior_mansion,exterior_monastery,exterior_mosque,exterior_motorcycle,exterior_mountains,exterior_ocean,exterior_office,exterior_palace,exterior_park,exterior_parkinglot,exterior_pier,exterior_plains,exterior_playground,exterior_polar,exterior_port,exterior_restaurant,exterior_river,exterior_road,exterior_ruins,exterior_school,exterior_sidewalk,exterior_sky,exterior_skyscraper,exterior_space,exterior_spacecraft,exterior_stadium,exterior_station_gas,exterior_station_subway,exterior_station_train,exterior_store,exterior_suburb,exterior_synagogue,exterior_temple,exterior_theater,exterior_town,exterior_train,exterior_truck,exterior_tunnel,exterior_warehouse,exterior_wetlands,interior,interior_airplane_cabin,interior_airplane_cockpit,interior_airport,interior_arena,interior_auditorium,interior_auto_repair_shop,interior_bar,interior_barn,interior_bathroom,interior_bedroom,interior_boat,interior_bus,interior_cafe,interior_cafeteria,interior_car,interior_cave,interior_classroom,interior_cloister,interior_closet,interior_command_center,interior_commercialkitchen,interior_conferenceroom,interior_courtroom,interior_crypt,interior_dancefloor,interior_diningroom,interior_dungeon,interior_elevator,interior_factory,interior_foyer,interior_gym,interior_hallway,interior_helicopter,interior_hospital,interior_kitchen,interior_livingroom,interior_lobby,interior_mall,interior_meditation,interior_nave,interior_office,interior_office_cubicle,interior_office_open,interior_prayer_hall,interior_prison,interior_pulpit,interior_restaurant,interior_spacecraft,interior_stage,interior_stairwell,interior_station_bus,interior_station_fire,interior_station_police,interior_station_subway,interior_station_train,interior_store,interior_store_aisle,interior_store_checkout,interior_study,interior_subway,interior_synagogue,interior_throneroom,interior_train,interior_truck,interior_warehouse,shot_angle_aerial,shot_angle_eyelevel,shot_angle_high,shot_angle_low,shot_focus_deep,shot_focus_out,shot_focus_shallow,shot_framing_closeup,shot_framing_extemelong,shot_framing_extremecloseup,shot_framing_long,shot_framing_medium,shot_level_level,shot_level_tilted,shot_lighting_hard,shot_lighting_key_high,shot_lighting_key_low,shot_lighting_silhouette,shot_lighting_soft,shot_subject_animal,shot_subject_location,shot_subject_object,shot_subject_person,shot_subject_person_body,shot_subject_person_face,shot_subject_person_feet,shot_subject_person_hands,shot_subject_text,shot_timeofday_day,shot_timeofday_night,shot_timeofday_twilight,shot_type_master,shot_type_overtheshoulder,shot_type_portrait,shot_type_twoshot,texture_banded,texture_blotchy,texture_braided,texture_bubbly,texture_bumpy,texture_chequered,texture_cobwebbed,texture_cracked,texture_crosshatched,texture_crystalline,texture_dotted,texture_fibrous,texture_flecked,texture_frilly,texture_gauzy,texture_grid,texture_grooved,texture_honeycombed,texture_interlaced,texture_knitted,texture_lacelike,texture_lined,texture_marbled,texture_matted,texture_meshed,texture_paisley,texture_perforated,texture_pitted,texture_pleated,texture_porous,texture_potholed,texture_scaly,texture_smeared,texture_spiralled,texture_sprinkled,texture_stained,texture_stratified,texture_striped,texture_studded,texture_swirly,texture_veined,texture_waffled,texture_woven,texture_wrinkled,texture_zigzaggedpath/to/image/-5.jpg,1,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1

我能够通过 tf.data.experimental.make_csv_dataset 将 CSV 加载到数据集中，如下所示：

# 创建我们的数据集BATCH_SIZE = 32FILE_PATH = ["filepath"]COLUMN_NAMES = ["filepath", "color_key_blue","color_key_green","color_key_luma","color_key_matte","color_saturation_desaturated","color_saturation_neutral","color_saturation_pastel","color_saturation_saturated","color_theory_analagous","color_theory_complementary","color_theory_monochrome","color_tones_blackwhite","color_tones_cool","color_tones_warm","exterior","exterior_airplane","exterior_airport","exterior_apartment","exterior_auto_body","exterior_beach","exterior_bicycle","exterior_boat","exterior_bridge","exterior_bus","exterior_bus_stop","exterior_canyon","exterior_car","exterior_castle","exterior_cathedral","exterior_cave","exterior_church","exterior_city","exterior_desert","exterior_farm","exterior_forest","exterior_glacier","exterior_helicopter","exterior_hospital","exterior_house","exterior_industrial","exterior_lake","exterior_library","exterior_mall","exterior_mansion","exterior_monastery","exterior_mosque","exterior_motorcycle","exterior_mountains","exterior_ocean","exterior_office","exterior_palace","exterior_park","exterior_parkinglot","exterior_pier","exterior_plains","exterior_playground","exterior_polar","exterior_port","exterior_restaurant","exterior_river","exterior_road","exterior_ruins","exterior_school","exterior_sidewalk","exterior_sky","exterior_skyscraper","exterior_space","exterior_spacecraft","exterior_stadium","exterior_station_gas","exterior_station_subway","exterior_station_train","exterior_store","exterior_suburb","exterior_synagogue","exterior_temple","exterior_theater","exterior_town","exterior_train","exterior_truck","exterior_tunnel","exterior_warehouse","exterior_wetlands","interior","interior_airplane_cabin","interior_airplane_cockpit","interior_airport","interior_arena","interior_auditorium","interior_auto_repair_shop","interior_bar","interior_barn","interior_bathroom","interior_bedroom","interior_boat","interior_bus","interior_cafe","interior_cafeteria","interior_car","interior_cave","interior_classroom","interior_cloister","interior_closet","interior_command_center","interior_commercialkitchen","interior_conferenceroom","interior_courtroom","interior_crypt","interior_dancefloor","interior_diningroom","interior_dungeon","interior_elevator","interior_factory","interior_foyer","interior_gym","interior_hallway","interior_helicopter","interior_hospital","interior_kitchen","interior_livingroom","interior_lobby","interior_mall","interior_meditation","interior_nave","interior_office","interior_office_cubicle","interior_office_open","interior_prayer_hall","interior_prison","interior_pulpit","interior_restaurant","interior_spacecraft","interior_stage","interior_stairwell","interior_station_bus","interior_station_fire","interior_station_police","interior_station_subway","interior_station_train","interior_store","interior_store_aisle","interior_store_checkout","interior_study","interior_subway","interior_synagogue","interior_throneroom","interior_train","interior_truck","interior_warehouse","shot_angle_aerial","shot_angle_eyelevel","shot_angle_high","shot_angle_low","shot_focus_deep","shot_focus_out","shot_focus_shallow","shot_framing_closeup","shot_framing_extemelong","shot_framing_extremecloseup","shot_framing_long","shot_framing_medium","shot_level_level","shot_level_tilted","shot_lighting_hard","shot_lighting_key_high","shot_lighting_key_low","shot_lighting_silhouette","shot_lighting_soft","shot_subject_animal","shot_subject_location","shot_subject_object","shot_subject_person","shot_subject_person_body","shot_subject_person_face","shot_subject_person_feet","shot_subject_person_hands","shot_subject_text","shot_timeofday_day","shot_timeofday_night","shot_timeofday_twilight","shot_type_master","shot_type_overtheshoulder","shot_type_portrait","shot_type_twoshot","texture_banded","texture_blotchy","texture_braided","texture_bubbly","texture_bumpy","texture_chequered","texture_cobwebbed","texture_cracked","texture_crosshatched","texture_crystalline","texture_dotted","texture_fibrous","texture_flecked","texture_frilly","texture_gauzy","texture_grid","texture_grooved","texture_honeycombed","texture_interlaced","texture_knitted","texture_lacelike","texture_lined","texture_marbled","texture_matted","texture_meshed","texture_paisley","texture_perforated","texture_pitted","texture_pleated","texture_porous","texture_potholed","texture_scaly","texture_smeared","texture_spiralled","texture_sprinkled","texture_stained","texture_stratified","texture_striped","texture_studded","texture_swirly","texture_veined","texture_waffled","texture_woven","texture_wrinkled","texture_zigzagged"]LABEL_NAMES = ["color_key_blue","color_key_green","color_key_luma","color_key_matte","color_saturation_desaturated","color_saturation_neutral","color_saturation_pastel","color_saturation_saturated","color_theory_analagous","color_theory_complementary","color_theory_monochrome","color_tones_blackwhite","color_tones_cool","color_tones_warm","exterior","exterior_airplane","exterior_airport","exterior_apartment","exterior_auto_body","exterior_beach","exterior_bicycle","exterior_boat","exterior_bridge","exterior_bus","exterior_bus_stop","exterior_canyon","exterior_car","exterior_castle","exterior_cathedral","exterior_cave","exterior_church","exterior_city","exterior_desert","exterior_farm","exterior_forest","exterior_glacier","exterior_helicopter","exterior_hospital","exterior_house","exterior_industrial","exterior_lake","exterior_library","exterior_mall","exterior_mansion","exterior_monastery","exterior_mosque","exterior_motorcycle","exterior_mountains","exterior_ocean","exterior_office","exterior_palace","exterior_park","exterior_parkinglot","exterior_pier","exterior_plains","exterior_playground","exterior_polar","exterior_port","exterior_restaurant","exterior_river","exterior_road","exterior_ruins","exterior_school","exterior_sidewalk","exterior_sky","exterior_skyscraper","exterior_space","exterior_spacecraft","exterior_stadium","exterior_station_gas","exterior_station_subway","exterior_station_train","exterior_store","exterior_suburb","exterior_synagogue","exterior_temple","exterior_theater","exterior_town","exterior_train","exterior_truck","exterior_tunnel","exterior_warehouse","exterior_wetlands","interior","interior_airplane_cabin","interior_airplane_cockpit","interior_airport","interior_arena","interior_auditorium","interior_auto_repair_shop","interior_bar","interior_barn","interior_bathroom","interior_bedroom","interior_boat","interior_bus","interior_cafe","interior_cafeteria","interior_car","interior_cave","interior_classroom","interior_cloister","interior_closet","interior_command_center","interior_commercialkitchen","interior_conferenceroom","interior_courtroom","interior_crypt","interior_dancefloor","interior_diningroom","interior_dungeon","interior_elevator","interior_factory","interior_foyer","interior_gym","interior_hallway","interior_helicopter","interior_hospital","interior_kitchen","interior_livingroom","interior_lobby","interior_mall","interior_meditation","interior_nave","interior_office","interior_office_cubicle","interior_office_open","interior_prayer_hall","interior_prison","interior_pulpit","interior_restaurant","interior_spacecraft","interior_stage","interior_stairwell","interior_station_bus","interior_station_fire","interior_station_police","interior_station_subway","interior_station_train","interior_store","interior_store_aisle","interior_store_checkout","interior_study","interior_subway","interior_synagogue","interior_throneroom","interior_train","interior_truck","interior_warehouse","shot_angle_aerial","shot_angle_eyelevel","shot_angle_high","shot_angle_low","shot_focus_deep","shot_focus_out","shot_focus_shallow","shot_framing_closeup","shot_framing_extemelong","shot_framing_extremecloseup","shot_framing_long","shot_framing_medium","shot_level_level","shot_level_tilted","shot_lighting_hard","shot_lighting_key_high","shot_lighting_key_low","shot_lighting_silhouette","shot_lighting_soft","shot_subject_animal","shot_subject_location","shot_subject_object","shot_subject_person","shot_subject_person_body","shot_subject_person_face","shot_subject_person_feet","shot_subject_person_hands","shot_subject_text","shot_timeofday_day","shot_timeofday_night","shot_timeofday_twilight","shot_type_master","shot_type_overtheshoulder","shot_type_portrait","shot_type_twoshot","texture_banded","texture_blotchy","texture_braided","texture_bubbly","texture_bumpy","texture_chequered","texture_cobwebbed","texture_cracked","texture_crosshatched","texture_crystalline","texture_dotted","texture_fibrous","texture_flecked","texture_frilly","texture_gauzy","texture_grid","texture_grooved","texture_honeycombed","texture_interlaced","texture_knitted","texture_lacelike","texture_lined","texture_marbled","texture_matted","texture_meshed","texture_paisley","texture_perforated","texture_pitted","texture_pleated","texture_porous","texture_potholed","texture_scaly","texture_smeared","texture_spiralled","texture_sprinkled","texture_stained","texture_stratified","texture_striped","texture_studded","texture_swirly","texture_veined","texture_waffled","texture_woven","texture_wrinkled","texture_zigzagged"]def get_dataset(file_path, **kwargs):  dataset = tf.data.experimental.make_csv_dataset(      file_path,      batch_size=BATCH_SIZE, # 人为设置小批量以便示例更易展示。      na_value="?",      num_epochs=1,      column_names=COLUMN_NAMES,      ignore_errors=True,       shuffle=True, # 临时设置      **kwargs)  return datasetcsv_dataset = get_dataset(csv_file_path)

然后我将数据集转换为返回一个包含 string tensors 和 tensors 的元组，其中包含每个标签的单维向量，包含我的 1、0 或 -1 的值，用于每个标签：

# 我们需要将我们的数据集拆分成匹配的文件路径和标签，作为每个标签的稀疏向量，# 包含 1、0、-1 值，表示标签概念存在、标签概念不存在、不知道是否存在 # 例如，CSV 行像 # FOLD_0/color_key_blue/1-5.jpg,1,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1# 应该返回：# * 一个包含路径的张量，如 data_root + FOLD_0/color_key_blue/1-5.jpg, # * 一个包含值 [1,0,0,0, -1 等] 的张量# 一个返回路径、有序字典中仅包含导入 tensorflow.python.utildef split_csv_to_path_and_labels(csv_row_Dict):  print("调用 split_csv_to_path_and_label")  filepath = csv_row_Dict.pop('filepath')  data_root_tensor = tf.constant(data_root)  filepath = tf.strings.join([data_root_tensor, filepath], separator='')  # 创建一个新的张量，其中包含 LABEL_NAMES 键的值，但打包成一个 0, len(LABEL_NAMES) 数组  labels = tf.stack(list(csv_row_Dict.values()), axis=1)  return filepath, labels# 从我们的 CSV 中创建一个新的数据集，通过映射每个值到上述函数split_dataset = csv_dataset.map(split_csv_to_path_and_labels)

最后，我需要创建一个数据集，为批次中的每个字符串张量加载图像，并返回一批图像/标签元组。我定义了一个加载图像的函数，如下所示：

def load_and_preprocess_image(batch_of_paths, batch_of_labels):  # 从我们的批次中解包我们的图像  list_of_paths = tf.unstack(batch_of_paths, num=BATCH_SIZE)  list_of_images = []  for path in list_of_paths:      image = tf.read_file(path)      if image == None:        print("无法加载路径上的图像：" + path )      image = preprocess_image(image)      list_of_images.append(image)  # 将我们现在加载并处理过的图像重新打包成一个批次  batch_of_images = tf.stack(list_of_images, 0)  return batch_of_images,batch_of_labels

然后通过映射数据集到这个函数来创建一个新的数据集：

image_and_labels_ds = split_dataset.map(load_and_preprocess_image, num_parallel_calls=AUTOTUNE)

由于上述函数中硬编码了 BATCH_SIZE，它一直工作到最后一个批次小于批量大小，然后 TF 报错，如上所示错误。

我如何解决这个问题！？

我查看了 TensorFlow 中的动态批量大小，但对我来说不清楚如何解决这个问题

回答：

我通过使用 tf.map_fn 并映射一个处理单个字符串张量并返回预处理（归一化）图像张量的函数来解决这个问题：

主要的 tf.map_fn 调用：

def load_and_preprocess_image_batch(batch_of_paths, batch_of_labels):    batch_of_images = tf.map_fn(load_and_preprocess_single_image_from_tensor, batch_of_paths, dtype=tf.float32)    return batch_of_images, batch_of_labels

以及 load_and_preprocess_single_image_from_tensor 的声明和我的预处理代码，供有兴趣的人参考：

IMG_SIZE = 224  def preprocess_image(image):  image = tf.image.decode_jpeg(image, channels=3)  image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])  image /= 255.0  # 归一化到 [0,1] 范围  return imagedef load_and_preprocess_single_image_from_tensor(path):   image = tf.read_file(path)   if image == None:    print("无法加载路径上的图像：" + path )   return preprocess_image(image) 

最后，确保我的数据集使用 load_and_preprocess_image_batch 函数：

image_and_labels_ds = split_dataset.map(load_and_preprocess_image_batch, num_parallel_calls=AUTOTUNE)