内部错误(请参阅上面的跟踪信息):Blas GEMM 启动失败

我用Keras训练了一个模型,并想对其进行评估。但我总是遇到这个错误。我在这里找到一个解决方案 TensorFlow: InternalError: Blas SGEMM launch failed,但这只是针对TensorFlow的解决方案。

Using TensorFlow backend.2017-11-01 10:40:49.120525: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.2017-11-01 10:40:49.120546: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.2017-11-01 10:40:49.120553: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.2017-11-01 10:40:49.120557: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.2017-11-01 10:40:49.120562: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.2017-11-01 10:40:49.266103: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero2017-11-01 10:40:49.266511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: name: GeForce 940MXmajor: 5 minor: 0 memoryClockRate (GHz) 1.189pciBusID 0000:01:00.0Total memory: 1.96GiBFree memory: 1.78GiB2017-11-01 10:40:49.266528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 2017-11-01 10:40:49.266534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y 2017-11-01 10:40:49.266542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0)x_train shape: (50000, 32, 32, 3)50000 train samples10000 test samples2017-11-01 10:40:54.162805: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED2017-11-01 10:40:54.162825: W tensorflow/stream_executor/stream.cc:1756] attempting to perform BLAS operation using StreamExecutor without BLAS supportTraceback (most recent call last):  File "/home/viktor/PycharmProjects/ProjectSSD/test.py", line 39, in <module>    scores = model.evaluate(x_test_bin, y_test, verbose=1)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/models.py", line 896, in evaluate    sample_weight=sample_weight)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1657, in evaluate    steps=steps)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1339, in _test_loop    batch_outs = f(ins_batch)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2273, in __call__    **self.session_kwargs)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run    run_metadata_ptr)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1124, in _run    feed_dict_tensor, options, run_metadata)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run    options, run_metadata)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call    raise type(e)(node_def, op, message)tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(32, 2304), b.shape=(2304, 512), m=32, n=512, k=2304     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]Caused by op u'dense_1/MatMul', defined at:  File "/home/viktor/PycharmProjects/ProjectSSD/test.py", line 13, in <module>    model = load_model(save_dir + '/' + model_name)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/models.py", line 239, in load_model    model = model_from_config(model_config, custom_objects=custom_objects)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/models.py", line 313, in model_from_config    return layer_module.deserialize(config, custom_objects=custom_objects)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize    printable_module_name='layer')  File "/home/viktor/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object    list(custom_objects.items())))  File "/home/viktor/.local/lib/python2.7/site-packages/keras/models.py", line 1214, in from_config    model.add(layer)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/models.py", line 475, in add    output_tensor = layer(self.outputs[0])  File "/home/viktor/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 602, in __call__    output = self.call(inputs, **kwargs)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/layers/core.py", line 841, in call    output = K.dot(inputs, self.kernel)  File "/home/viktor/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 998, in dot    out = tf.matmul(x, y)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1844, in matmul    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 1289, in _mat_mul    transpose_b=transpose_b, name=name)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op    op_def=op_def)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op    original_op=self._default_original_op, op_def=op_def)  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-accessInternalError (see above for traceback): Blas GEMM launch failed : a.shape=(32, 2304), b.shape=(2304, 512), m=32, n=512, k=2304     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

这是代码:首先加载模型。加载数据集。准备数据进行评估。然后进行评估。

from __future__ import print_functionimport kerasfrom keras.datasets import cifar10from keras.models import load_modelimport numpy as npimport osnum_classes = 10save_dir = os.path.join(os.getcwd(), 'examples/saved_models')model_name = 'keras_cifar10_trained_model.h5'model = load_model(save_dir + '/' + model_name)# The data, shuffled and split between train and test sets:(x_train, y_train), (x_test, y_test) = cifar10.load_data()print('x_train shape:', x_train.shape)print(x_train.shape[0], 'train samples')print(x_test.shape[0], 'test samples')# Convert class vectors to binary class matrices.y_train = keras.utils.to_categorical(y_train, num_classes)y_test = keras.utils.to_categorical(y_test, num_classes)x_train_float = x_train.astype('float32')x_test_float = x_test.astype('float32')x_train_bin = x_train_float / 255x_test_bin = x_test_float / 255# Score trained model.scores = model.evaluate(x_test_bin, y_test, verbose=1)print('Test loss:', scores[0])print('Test accuracy:', scores[1])

我发现其他人也有同样的问题,如果他们同时运行两个会话。当执行以下代码行时:

model = load_model(save_dir + '/' + model_name)

GPU使用率会增加。(使用watch -n 0.5 nvidia-smi监控)这可能是问题所在吗?


回答:

@[隐藏人名],对不起浪费了你的时间。

错误来自代码的一部分,与cv2有关。我已经开启了一个新问题。

cv2, keras, InternalError (see above for traceback): Blas GEMM launch failed

非常感谢你,你帮助我更接近解决方案了

Related Posts

使用LSTM在Python中预测未来值

这段代码可以预测指定股票的当前日期之前的值,但不能预测…

如何在gensim的word2vec模型中查找双词组的相似性

我有一个word2vec模型,假设我使用的是googl…

dask_xgboost.predict 可以工作但无法显示 – 数据必须是一维的

我试图使用 XGBoost 创建模型。 看起来我成功地…

ML Tuning – Cross Validation in Spark

我在https://spark.apache.org/…

如何在React JS中使用fetch从REST API获取预测

我正在开发一个应用程序,其中Flask REST AP…

如何分析ML.NET中多类分类预测得分数组?

我在ML.NET中创建了一个多类分类项目。该项目可以对…

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注