在Azure ML服务上部署时,PySpark ALSModel加载失败,出现错误java.util.NoSuchElementException: Param blockSize不存在

我试图在Azure ML服务上部署使用PySpark训练的ALS模型。我提供了一个score.py文件,该文件使用ALSModel.load()函数加载训练模型。以下是我score.py文件的代码。

import osfrom azureml.core.model import Modelfrom pyspark.ml.recommendation import ALS, ALSModelfrom pyspark.sql.types import StructType, StructFieldfrom pyspark.sql.types import DoubleType, StringTypefrom pyspark.sql import SQLContextfrom pyspark import SparkContextsc = SparkContext.getOrCreate()sqlContext = SQLContext(sc)spark = sqlContext.sparkSessioninput_schema = StructType([StructField("UserId", StringType())])reader = spark.readreader.schema(input_schema)def init():    global model    # 注意这里的"iris.model"是在工作区下注册的模型名称    # 此调用应返回本地磁盘上model.pkl文件的路径。    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), "recommendation-model")    # 将模型文件加载回LogisticRegression模型    model = ALSModel.load(model_path)    def run(data):    try:        input_df = reader.json(sc.parallelize([data]))        input_df = indexer.transform(input_df)                res = model.recommendForUserSubset(input_df[['UserId_index']], 10)        # 只要是JSON可序列化的数据类型,您都可以返回        return result.collect()[0]['recommendations']    except Exception as e:        traceback.print_exc()        error = str(e)        return error

当我使用Azure ML服务中的Model.deploy函数将其部署为LocalWebService时,我得到以下错误

Generating Docker build context.Package creation SucceededLogging into Docker registry viennaglobal.azurecr.ioLogging into Docker registry viennaglobal.azurecr.ioBuilding Docker image from Dockerfile...Step 1/5 : FROM viennaglobal.azurecr.io/azureml/azureml_43542b56c5ec3e8d0f68e1556558411f ---> 5b3bb174ca5fStep 2/5 : COPY azureml-app /var/azureml-app ---> 8e540c0746f7Step 3/5 : RUN mkdir -p '/var/azureml-app' && echo eyJhY2NvdW50Q29udGV4dCI6eyJzdWJzY3JpcHRpb25JZCI6IjNkN2M1ZjM4LTI1ODEtNGUxNi05NTdhLWEzOTU1OGI1ZjBiMyIsInJlc291cmNlR3JvdXBOYW1lIjoiZGV2LW9tbmljeC10ZnMtYWkiLCJhY2NvdW50TmFtZSI6ImRldi10ZnMtYWktd29ya3NwYWNlIiwid29ya3NwYWNlSWQiOiI1NjkzNGMzNC1iZmYzLTQ3OWUtODRkMy01OGI4YTc3ZTI4ZjEifSwibW9kZWxzIjp7fSwibW9kZWxzSW5mbyI6e319 | base64 --decode > /var/azureml-app/model_config_map.json ---> Running in 502ad8edf91e ---> a1bc5e0283d0Step 4/5 : RUN mv '/var/azureml-app/tmpvxhomyin.py' /var/azureml-app/main.py ---> Running in eb4ec1a0b702 ---> 6a3296fe6420Step 5/5 : CMD ["runsvdir","/var/runit"] ---> Running in 834fd746afef ---> 5b9f8be538c0Successfully built 5b9f8be538c0Successfully tagged recommend-service:latestContainer (name:musing_borg, id:0f3163692f5119685eee5ed59c8e00aa96cd472f765e7db67653f1a6ce852e83) cannot be killed.Container has been successfully cleaned up.Image sha256:0f146f4752878bbbc0e876f4477cc2877ff12a366fca18c986f9a9c2949d028b successfully removed.Starting Docker container...Docker container running.Checking container health...ERROR - Error: Container has crashed. Did your init method fail?Container Logs:/bin/bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by /bin/bash)/bin/bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by /bin/bash)/bin/bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by /bin/bash)/bin/bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by /bin/bash)2020-07-30T11:57:00,312735664+00:00 - rsyslog/run 2020-07-30T11:57:00,312768364+00:00 - gunicorn/run 2020-07-30T11:57:00,313017966+00:00 - iot-server/run bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by bash)2020-07-30T11:57:00,313969073+00:00 - nginx/run /usr/sbin/nginx: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting.../bin/bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by /bin/bash)2020-07-30T11:57:00,597835804+00:00 - iot-server/finish 1 02020-07-30T11:57:00,598826211+00:00 - Exit code 1 is normal. Not restarting iot-server.Starting gunicorn 19.9.0Listening at: http://127.0.0.1:31311 (10)Using worker: syncworker timeout is set to 300Booting worker with pid: 41bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by bash)bash: /azureml-envs/azureml_7fbe163ce1d4208cd897650a64b7a54d/lib/libtinfo.so.5: no version information available (required by bash)

然而,在Jupyter笔记本中执行时,ALSModel.load()可以正常工作。


回答:

需要检查的几件事:

  1. 您的模型是否已在工作区中注册?AZUREML_MODEL_DIR仅适用于已注册的模型。有关注册模型的信息,请参见此链接
  2. 您在InferenceConfig中指定的pyspark.ml.recommendation版本是否与您本地使用的版本相同?这种错误可能是由于版本差异引起的
  3. 您是否查看过print(service.get_logs())的输出?请查看我们的故障排除和调试文档,了解您可以尝试的其他方法

Related Posts

使用LSTM在Python中预测未来值

这段代码可以预测指定股票的当前日期之前的值,但不能预测…

如何在gensim的word2vec模型中查找双词组的相似性

我有一个word2vec模型,假设我使用的是googl…

dask_xgboost.predict 可以工作但无法显示 – 数据必须是一维的

我试图使用 XGBoost 创建模型。 看起来我成功地…

ML Tuning – Cross Validation in Spark

我在https://spark.apache.org/…

如何在React JS中使用fetch从REST API获取预测

我正在开发一个应用程序,其中Flask REST AP…

如何分析ML.NET中多类分类预测得分数组?

我在ML.NET中创建了一个多类分类项目。该项目可以对…

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注