我正在加载一个包含多个输入图像的数据集。输入图像路径应该只在批次时间解码，以便处理大型数据集。
该数据集有N个图像路径输入和M个浮点数输出。每组输入的图像具有不同的分辨率。
Data is ([img_input_1.png, img_input_2.png, ...], [0.65, 0.7, 0.8])模型使用Keras的函数式API在符号模式下运行。
这是最近编辑的代码
from itertools import zip_longestdef read_image(path, shape): try: image = tf.io.read_file(path) image = tf.image.decode_png(image) image = tf.image.resize(image, [shape[1],shape[2]]) image /= 255.0 return image except: print('ERROR: preprocess_image: bad path', path) def load_image(x, y, shp): pout = [(k, x[k]) for k in x.keys()] l1 = tf.convert_to_tensor(list(x)) l2 = tf.convert_to_tensor(list(x.values())) pl = tf.map_fn( lambda args: (read_image(args[0], shp), args[1]), [l1, l2], dtype=(tf.float32, tf.float32) ) pl = {path: (pl[0][i], pl[1][i]) for i, path in enumerate(x)} return (pl,y)def dataset_prep(json_data, seq, batch_size): # LOAD DATA FROM JSON x,y = json_parse_x_y(json_data[seq]) xx = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...} yy = [*zip_longest(*y)] # GET SHAPES (hard coded atm) lns = [[len(xxx)] for xxx in xx] rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx] shp = [*zip_longest(*[lns,rzs])] shp = [list(s) for s in shp] shp = [[*itertools.chain.from_iterable(s)] for s in shp] xd = dict([[ "input_{}".format(i+1),np.array(y)] for i,y in [*enumerate(xx)]]) yd = dict([["output_{}".format(i+1),np.array(y)] for i,y in [*enumerate(yy)]]) ds = tf.data.Dataset.from_tensor_slices((xd, yd)) ds = ds.shuffle(10000) ds = ds.repeat() ds = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE) ds = ds.batch(batch_size) if batch_size else ds ds = ds.prefetch(AUTOTUNE) return ds这是我遇到的错误：
Traceback (most recent call last): File "/home/me/.local/bin/wavfeat", line 11, in <module> load_entry_point('wavfeat==0.1.0', 'console_scripts', 'wavfeat')() File "/home/me/.local/lib/python3.6/site-packages/wavfeat/__main__.py", line 91, in main analysis_batch_sql(obj) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in analysis_batch_sql qy = [*map(lambda c: run_elm(c[0], c[1]), ch)] File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in <lambda> qy = [*map(lambda c: run_elm(c[0], c[1]), ch)] File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 23, in run_elm out = fn(input, elm) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 196, in one_sec_onset_train return train(input, elm) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 182, in train ts = dataset_prep(jd, 'train', bc) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 123, in dataset_prep ds = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1146, in map self, map_func, num_parallel_calls, preserve_cardinality=True) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3264, in __init__ use_legacy_function=use_legacy_function) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2591, in __init__ self._function = wrapper_fn._get_concrete_function_internal() File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1366, in _get_concrete_function_internal *args, **kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1360, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1648, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1541, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 716, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2585, in wrapper_fn ret = _wrapper_helper(*args) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2530, in _wrapper_helper ret = func(*nested_args) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 123, in <lambda> ds = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 91, in load_image print("x['input_1'].values(): ", x['input_1'].values())AttributeError: 'Tensor' object has no attribute 'values'我做了什么导致路径无法加载？
编辑：
尝试pandrey的修复，我遇到了输入错误。这是在from_tensor_slices和ds.map之前的数据，然后是之后的数据：
pre_from_tensor_slices x: {'input_1': array(['/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/7388_39216_30--id=7388__sql_table=oac_1__sql_idx=405167__pitch=30__onset=39216.png', '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/2447_864_27--id=2447__sql_table=oac_1__sql_idx=415458__pitch=27__onset=864.png', '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/2386_20208_38--id=2386__sql_table=oac_1__sql_idx=433248__pitch=38__onset=20208.png', ..., '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/6261_24528_57--id=6261__sql_table=oac_1__sql_idx=449753__pitch=57__onset=24528.png', '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/3727_22944_31--id=3727__sql_table=oac_1__sql_idx=407620__pitch=31__onset=22944.png', '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/1668_7056_60--id=1668__sql_table=oac_1__sql_idx=381152__pitch=60__onset=7056.png'], dtype='<U162'), 'input_2': array(['/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/7388_39216_30--id=7388__sql_table=oac_1__sql_idx=405167__pitch=30__onset=39216.png', '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/2447_864_27--id=2447__sql_table=oac_1__sql_idx=415458__pitch=27__onset=864.png', '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/2386_20208_38--id=2386__sql_table=oac_1__sql_idx=433248__pitch=38__onset=20208.png', ..., '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/6261_24528_57--id=6261__sql_table=oac_1__sql_idx=449753__pitch=57__onset=24528.png', '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/3727_22944_31--id=3727__sql_table=oac_1__sql_idx=407620__pitch=31__onset=22944.png', '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/1668_7056_60--id=1668__sql_table=oac_1__sql_idx=381152__pitch=60__onset=7056.png'], dtype='<U162')}pre_from_tensor_slices y: {'output_1': array([0.817, 0.018, 0.421, ..., 0.511, 0.478, 0.147])}_________________________y: {'output_1': <tf.Tensor 'args_2:0' shape=() dtype=float64>}x: {'input_1': <tf.Tensor 'args_0:0' shape=() dtype=string>, 'input_2': <tf.Tensor 'args_1:0' shape=() dtype=string>}x.values(): dict_values([<tf.Tensor 'args_0:0' shape=() dtype=string>, <tf.Tensor 'args_1:0' shape=() dtype=string>])x['input_1']: Tensor("args_0:0", shape=(), dtype=string)运行x[‘input_1′].values()会抛出错误：’Tensor’ object has no attribute ‘values’
我在map_fn周围遇到了错误
File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 284, in _constant_impl allow_broadcast=allow_broadcast)) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 455, in make_tensor_proto raise ValueError("None values not supported.")ValueError: None values not supported.编辑2
尝试最新版本，我得到了以下错误
Traceback (most recent call last): File "/home/me/.local/bin/wavfeat", line 11, in <module> load_entry_point('wavfeat==0.1.0', 'console_scripts', 'wavfeat')() File "/home/me/.local/lib/python3.6/site-packages/wavfeat/__main__.py", line 91, in main analysis_batch_sql(obj) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in analysis_batch_sql qy = [*map(lambda c: run_elm(c[0], c[1]), ch)] File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in <lambda> qy = [*map(lambda c: run_elm(c[0], c[1]), ch)] File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 23, in run_elm out = fn(input, elm) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 216, in one_sec_onset_train return train(input, elm) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 203, in train vs = validation_prep(jd, 'validation', bc) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 176, in validation_prep ds = ds.map(map_func=load_and_preprocess_from_path_label, num_parallel_calls=AUTOTUNE) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1146, in map self, map_func, num_parallel_calls, preserve_cardinality=True) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3264, in __init__ use_legacy_function=use_legacy_function) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2591, in __init__ self._function = wrapper_fn._get_concrete_function_internal() File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1366, in _get_concrete_function_internal *args, **kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1360, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1648, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1541, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 716, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2585, in wrapper_fn ret = _wrapper_helper(*args) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2530, in _wrapper_helper ret = func(*nested_args) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 47, in load_and_preprocess_from_path_label pl = dict([(pk, tf.map_fn(load_and_preprocess_image, po, dtype=tf.float32)) for pk,po in pout]) File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 47, in <listcomp> pl = dict([(pk, tf.map_fn(load_and_preprocess_image, po, dtype=tf.float32)) for pk,po in pout]) File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/ops/map_fn.py", line 214, in map_fn raise ValueError("elems must be a 1+ dimensional Tensor, not a scalar")ValueError: elems must be a 1+ dimensional Tensor, not a scalar回答：
附加：不使用字典结构
这是实现您尝试的内容而不使用字典结构的完整代码（除了定义json_parse_x_y和声明AUTOTUNE之外）。
我测试了make_dataset是有效的（见下面的示例），所以如果您遇到问题，应该是由于load_tensors的规格错误所致。
from itertools import zip_longestimport tensorflow as tf# additionnally, `json_parse_x_y` must be defined# and `AUTOTUNE` must be declared (in my example, I set it to 2)def read_image(path, shape): """Read an image of givent filepath and tensor shape. Return a float tensor of given shape. """ try: image = tf.io.read_file(path) image = tf.image.decode_png(image) image = tf.image.resize(image, [shape[1], shape[2]]) image /= 255.0 return image except: raise FileNotFoundError("preprocess_image: bad path '%s'" % path)def load_images(paths, shapes): """Load an ensemble of images (associated with a single sample). paths : rank-1 string Tensor shapes : list of images' shapes (same length as `paths`) Return a tuple of float tensors containing the loaded images. """ return tuple(( read_image(paths[i], shapes[i]) for i in range(len(shapes)) ))def load_tensors(json_data, seq): """Load images descriptors from a json dump. Return a tuple containing: * a rank-2 tensor containing lists of image paths (str) * a rank-2 tensor containing resolution values (float) * a list of image shapes, of same length as the rank-2 tensor's second axis """ x,y = json_parse_x_y(json_data[seq]) xx = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...} yy = [*zip_longest(*y)] # GET SHAPES (hard coded atm) lns = [[len(xxx)] for xxx in xx] rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx] shp = [*zip_longest(*[lns,rzs])] shp = [list(s) for s in shp] shp = [[*itertools.chain.from_iterable(s)] for s in shp] return (xx, yy, shp)def make_dataset(xx, yy, shp, batch_size): """Build a Dataset instance containing loaded images. xx, yy, shp : see the specification of `load_tensors`'s outputs batch_size : batch size to set on the Dataset Return a Dataset instance where each batched sample is a tuple containing two elements: first, a tuple containing N loaded images' rank-3 tensors; second, a rank-1 tensor containing M float values. (to be clear: batching adds a dimension to all those tensors) """ data = tf.data.Dataset.from_tensor_slices((xx, yy)) data = data.shuffle(10000) data = data.map(lambda x, y: (load_images(x, shapes), y)) data = data.repeat() data = data.batch(batch_size) if batch_size else data data = data.prefetch(AUTOTUNE) return datadef dataset_prep(json_data, seq, batch_size): """Full pipeline to making a Dataset from json.""" xx, yy, shapes = load_tensors(json_data, seq) return make_dataset(xx, yy, shapes)示例，使用“手工制作”的值；所有图像实际上是这张经典图像，形状为[512, 512, 3]。
import numpy as npimport tensorflow as tf# import previous code# Here, N = 2, and I make 2 samples.x = tf.convert_to_tensor(np.array([ ['image_1a.png', 'image_1b.png'], ['image_2a.png', 'image_2b.png']]))shapes = [[1, 512, 512], [1, 512, 512]] # images are initially [512, 512, 3]# Here, M = 3, and I make 2 samples. Values are purely random.y = tf.convert_to_tensor(np.array([ [.087, .92, .276], [.242, .37, .205]]))# This should work.data = make_dataset(x, y, shapes, batch_size=1)# Output signature is <PrefetchDataset shapes:# (((None, 512, 512, None), (None, 512, 512, None)), (None, 3)),# types: ((tf.float32, tf.float32), tf.float64)# ># Where the first None is actually `batch_size`# and the second is, in this case, 3.当前问题的回答：
好的，您现在遇到的问题是修订后的load_image函数不符合数据集的规格，因此引发了异常。请在下面找到一个经过全面编辑且似乎有效的代码（我使用自定义图像在我的计算机上进行了测试，将xd / yd字典初始化为看起来像您报告的x和y在数据集中的张量）。它不是很美观，我个人建议放弃字典结构，但它有效：
from itertools import zip_longestdef read_image(path, shape): try: image = tf.io.read_file(path) image = tf.image.decode_png(image) image = tf.image.resize(image, [shape[1],shape[2]]) image /= 255.0 return image except: raise FileNotFoundError("preprocess_image: bad path '%s'" % path)# CHANGED: load_image is actually uselessdef dataset_prep(json_data, seq, batch_size): # LOAD DATA FROM JSON x,y = json_parse_x_y(json_data[seq]) xx = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...} yy = [*zip_longest(*y)] # GET SHAPES (hard coded atm) lns = [[len(xxx)] for xxx in xx] rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx] shp = [*zip_longest(*[lns,rzs])] shp = [list(s) for s in shp] shp = [[*itertools.chain.from_iterable(s)] for s in shp] xd = dict([[ "input_{}".format(i+1),np.array(y)] for i,y in [*enumerate(xx)]]) yd = dict([["output_{}".format(i+1),np.array(y)] for i,y in [*enumerate(yy)]]) ds = tf.data.Dataset.from_tensor_slices((xd, yd)) ds = ds.shuffle(10000) # CHANGED: the following line, to run images import (also moved epeat instruction later) ds = ds.map( lambda x, y: ( {key: read_image(path, shp[i]) for i, (key, path) in enumerate(x.items())}, y ), num_parallel_calls=AUTOTUNE ) ds = ds.repeat() ds = ds.batch(batch_size) if batch_size else ds ds = ds.prefetch(AUTOTUNE) return ds初始回答（在问题编辑之前）：
在本回答中，我只处理load_image引发的异常，但可能还需要对其余部分进行一些工作——我没有进行测试，因为手头没有方便的数据集。
异常消息实际上相当明确：您将一个标量元素（例如[(k, tf.map_fn(lambda x: read_image(x, shp), n, dtype=tf.float32)) for k,n in pout]中的n）作为elems参数传递给tf.map_fn，而它期望一个张量（或（可能嵌套的）列表或张量元组），如其文档中明确规定。
您还错误地使用了tf.map_fn在引用的代码行中，因为基本上您将其与Python的意图列表混淆了，而您应该使用其中之一或另一个。
使用意图列表（同时替换load_image函数中之前无用的行）：pl = {path: (load_image(path, shp), res) for path, res in x.items()}
使用tf.map_fn：
# Read all images, return two tensors, one with images, the other with resolutions.# (so, resolutions inclusion in this is actually useless and should be redesigned)pl = tf.map_fn( lambda args: (read_image(args[0], shp), args[1]), [tf.convert_to_tensor(list(x)), tf.convert_to_tensor(list(x.values()))], dtype=(tf.float32, tf.float32))# If you really, really want to return a dict, but is it an optimal design?pl = {path: (pl[0][i], pl[1][i]) for i, path in enumerate(x)}我不知道以这种方式返回字典是否是最优的（甚至与数据集实例化兼容），然而，如果您的代码其余部分正常工作，这应该可以解决问题。
无论如何，如果您想遍历一个字典，请继续使用第一个版本或第二个版本的修改版（这可能具有并行化读取图像的优势）。
希望这对您有帮助 🙂
