Inference deployment of deep learning models refers to the process of applying trained deep learning models to actual scenarios for real-time prediction or inference. The following are general steps and related considerations for deep learning model inference deployment:
Model selection and training: First select a deep learning model suitable for the task and train it using labeled data. Common deep learning frameworks include TensorFlow, PyTorch, etc.
Model optimization and compression: In order to improve the efficiency and performance of the model during the deployment phase, the model can be optimized and compressed. For example, pruning (Pruning) can reduce unnecessary parameters and connections in the model; Quantization (Quantization) can reduce the numerical precision in the model, thereby reducing the size of the model and the amount of calculation; Distillation (Knowledge Distillation) can transfer the knowledge of the main model by using a smaller model.
Inference engine selection: Select a suitable inference engine based on specific needs and platform constraints. Commonly used inference engines include TensorRT, OpenVINO, ONNX Runtime, etc. These engines are optimized for different hardware devices to provide efficient model inference capabilities.
Hardware platform selection: According to the requirements of the inference engine, select a suitable hardware platform to run the deep learning model. Common hardware platforms include CPUs, GPUs, FPGAs, and dedicated deep learning processors.
Model deployment and integration: Deploy optimized deep learning models into target environments and integrate with other systems or applications. You can use the API provided by the framework or write custom code to realize the call and integration of the model.
Performance optimization and acceleration: During the deployment phase, the performance of the model can be optimized and accelerated according to actual needs. For example, reduce data transmission and computing overhead through batch inference (Batch Inference); use asynchronous inference (Asynchronous Inference) to improve concurrent processing capabilities.
Model update and maintenance: Models are regularly updated and maintained to maintain their performance and accuracy. Retraining or fine-tuning on new data may be required, along with versioning and model migration.
In the process of deep learning model inference deployment, factors such as data privacy, security, performance monitoring, and error handling also need to be considered to ensure the effective operation and reliability of the model. In addition, for large-scale deployment, containerization technologies such as Docker can be used to better manage and scale the inference service of the model.
Here we mainly introduce TensorFlow Servicing, OpenVINO, TensorRT, ONNX Runtime, and TFLite as examples.
一、TensorFlow Servicing
TensorFlow Serving is an open source framework for deploying trained TensorFlow models for reasoning. It provides a high-performance, scalable, and reliable service that makes it easy to deploy and manage machine learning models in production environments.
Below are some key features and capabilities of TensorFlow Serving:
Support multiple model formats: TensorFlow Serving supports multiple common model formats, including SavedModel, model exported by tf.estimator, Keras HDF5 model, etc. This enables users to conveniently use different model formats for inference deployment.
High-performance model loading and inference: TensorFlow Serving uses an efficient model loading and inference mechanism to achieve low-latency model inference services by preloading the model and keeping it in memory. In addition, it also supports multi-threading and asynchronous request processing to improve concurrent processing capabilities.
Flexible model version management: TensorFlow Serving allows multiple model versions to be deployed at the same time, and provides flexible model version control and management functions. In this way, users can easily perform A/B testing, incremental upgrades, and rollback operations to meet the needs of model updates and deployments.
Distributed system support: TensorFlow Serving supports distributed system architecture, which can distribute reasoning workloads to multiple computing nodes to achieve high concurrency and high availability. Users can perform horizontal expansion and load balancing as needed to meet the needs of large-scale production environments.
RESTful API and gRPC interface: TensorFlow Serving provides a RESTful API and a gRPC interface, enabling model reasoning requests to be made through HTTP or RPC. In this way, users can easily integrate with other systems or applications, and provide cross-platform service capabilities.
Monitoring and logging: TensorFlow Serving has built-in monitoring and logging capabilities to track model usage, performance metrics, and error messages in real time. This helps users in performance optimization, troubleshooting and system maintenance.
In summary, TensorFlow Serving provides a powerful and flexible framework that makes it simple and efficient to deploy a trained TensorFlow model to a production environment. Its features include support for multiple model formats, high-performance model loading and inference, flexible model versioning, distributed system support, multiple interface choices, and monitoring and logging capabilities. These features make TensorFlow Serving one of the preferred frameworks for machine learning model inference deployment.
Next, take handwritten digit recognition as an example to give an example code:
1、训练模型：
首先，需要训练并保存一个手写数字识别模型。这里假设已经通过TensorFlow训练好了一个模型，并保存在/path/to/model路径下。
2、安装TensorFlow Serving：
安装TensorFlow Serving可以通过以下命令完成：
pip install tensorflow-serving-api
3、启动TensorFlow Serving服务器： 使用以下命令启动TensorFlow Serving服务器，并指定要加载的模型路径：
$ tensorflow_model_server --port=8501 --model_name=handwritten_digit --model_base_path=/path/to/model/
其中，--port指定服务器的端口号，--model_name指定模型的名称，--model_base_path指定模型的存储路径。
4、客户端代码： 使用Python编写一个客户端脚本来进行推理请求。以下是一个简单的示例代码：
import requests
import numpy as np
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 构建请求的JSON数据
data = json.dumps({"signature_name": "serving_default", "instances": [image.tolist()]})
# 发送推理请求
headers = {"content-type": "application/json"}
response = requests.post('http://localhost:8501/v1/models/handwritten_digit:predict', data=data, headers=headers)
# 解析推理结果
predictions = json.loads(response.text)['predictions']
predicted_label = np.argmax(predictions[0])
print("Predicted Label:", predicted_label)
2、OpenVINO
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open source tool suite developed by Intel for inference deployment of deep learning models. It provides high-performance, low-latency and cross-platform solutions to help users deploy and optimize deep learning models on a variety of hardware devices.
The following are some key features and functions of the OpenVINO framework:
Model optimization: OpenVINO optimizes deep learning models by using specialized tools to improve inference performance. These include techniques such as model compression, quantization, and pruning to reduce model size, reduce computation, and improve model efficiency.
Hardware acceleration: OpenVINO supports a variety of Intel hardware accelerators, such as Intel CPU, VPU (Vision Processing Unit) and FPGA. It is optimized for these hardware devices to maximize their potential for deep learning inference.
Cross-platform compatibility: The OpenVINO framework provides support for different operating systems (such as Windows, Linux) and hardware platforms (such as x86 architecture, ARM architecture), enabling users to deploy and run deep learning models in different environments.
Multiple inference engines: The OpenVINO framework includes multiple inference engines, such as TensorFlow, Caffe, MXNet, etc. These engines can be deployed by converting the model to an OpenVINO-specific intermediate representation (IR), and leverage hardware accelerators for efficient model inference.
Support for asynchronous inference: The OpenVINO framework supports asynchronous inference, allowing multiple inference requests to be processed at the same time, improving concurrent performance and throughput. This is especially important for real-time application scenarios.
Integrated tools and libraries: OpenVINO provides a series of tools and libraries for tasks such as model conversion, performance analysis, model optimization, and deployment. For example, Model Optimizer is used to convert deep learning models to OpenVINO IR format, and Inference Engine is used to perform model inference.
Support multiple languages and interfaces: OpenVINO supports multiple programming languages and interfaces, such as C++, Python, Java, and RESTful API, etc. This allows users to develop and integrate applications using their familiar programming languages.
To sum up, OpenVINO is a powerful deep learning model inference deployment framework, featuring model optimization, hardware acceleration, cross-platform compatibility, multiple inference engines, asynchronous inference, integrated tools and libraries, and multiple languages and interfaces. It provides users with a high-performance, low-latency, and flexible solution that enables deep learning models to be deployed and run quickly and efficiently on a variety of hardware devices.
Next, take handwritten digit recognition as an example, and give an example code implementation:
1、准备模型：
首先，需要准备一个训练好的手写数字识别模型，并将其保存为TensorFlow SavedModel格式。
2、安装OpenVINO Toolkit：
在开始之前，请确保已经按照官方文档中的指引安装了OpenVINO Toolkit，并设置了必要的环境变量。
3、转换模型：
使用OpenVINO提供的Model Optimizer工具将TensorFlow模型转换为OpenVINO的中间表示形式。打开终端并执行以下命令：
$ python <path_to_openvino>/deployment_tools/model_optimizer/mo.py --input_model /path/to/model/saved_model.pb --model_name handwritten_digit --output_dir /path/to/output_directory
4、推理代码： 使用OpenVINO的Python API编写一个进行推理的脚本。以下是一个简单的示例代码：
from openvino.inference_engine import IECore
import numpy as np
# 加载OpenVINO的推理引擎
ie = IECore()
net = ie.read_network(model='path/to/output_directory/handwritten_digit.xml', weights='path/to/output_directory/handwritten_digit.bin')
exec_net = ie.load_network(network=net, device_name='CPU')
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 对输入数据进行预处理
preprocessed_image = preprocess(image)
# 执行推理
outputs = exec_net.infer(inputs={'input_blob_name': preprocessed_image})
# 解析推理结果
output_data = outputs['output_blob_name']
predicted_label = np.argmax(output_data)
print("Predicted Label:", predicted_label)
3、ONNX Runtime
ONNX Runtime is a high-performance open source framework for deep learning model inference deployment, which supports rapid deployment and running of trained models on different hardware platforms. The following is a detailed introduction of ONNX Runtime:
Support for multiple model formats: ONNX Runtime supports the Open Neural Network Exchange (ONNX) format, an interoperable model representation across multiple deep learning frameworks. By supporting the ONNX format, ONNX Runtime can seamlessly integrate with multiple deep learning frameworks (such as PyTorch, TensorFlow, etc.) and perform model inference.
Cross-platform compatibility: ONNX Runtime provides extensive support for multiple hardware platforms and operating systems, including CPUs, GPUs, and edge devices. It can run on operating systems such as Windows, Linux, and macOS, and is optimized for different hardware platforms.
High-performance reasoning engine: ONNX Runtime has a built-in high-efficiency reasoning engine, featuring low latency and high throughput. It uses a variety of optimization techniques, such as image optimization, automatic batch processing, parallel computing, etc., to maximize the performance of hardware devices.
Model optimization and conversion: ONNX Runtime provides some tools and APIs for model optimization and conversion. For example, the tools in ONNX Runtime's Model Zoo can be used to optimize, quantize, and prune the model to reduce model size and improve inference performance.
Dynamic graph support: ONNX Runtime supports dynamic graph models (such as PyTorch) and static graph models (such as TensorFlow), allowing users to seamlessly switch between these two models and achieve cross-framework interoperability.
Lightweight and Embeddable: ONNX Runtime is a lightweight framework with a small memory footprint and binary size. This makes it suitable for deployment to resource-constrained devices such as mobile devices or embedded systems.
Community support and activity: ONNX Runtime is an open source project with huge community support and an active developer community. This means users can benefit from extensive resources, documentation and examples, as well as receive timely technical support.
In summary, ONNX Runtime is a powerful deep learning model reasoning deployment framework, featuring interoperability across multiple deep learning frameworks, cross-platform compatibility, high-performance reasoning engine, model optimization and conversion, dynamic graph support, lightweight and embeddability. It provides users with a flexible and efficient solution for deploying and running trained deep learning models on various hardware devices.
Here also take handwritten digit recognition as an example, and give an example operation code:
1、准备模型：
首先，需要准备一个训练好的手写数字识别模型，并将其保存为ONNX格式。
2、安装ONNX Runtime：
在开始之前，请确保已经安装了ONNX Runtime库。可以使用以下命令安装：
pip install onnxruntime
3、加载并执行模型： 使用ONNX Runtime的Python API编写一个进行推理的脚本。以下是一个简单的示例代码：
import onnxruntime
import numpy as np
# 加载和初始化模型
sess = onnxruntime.InferenceSession('/path/to/model/handwritten_digit.onnx')
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 对输入数据进行预处理
preprocessed_image = preprocess(image)
# 执行推理
inputs = {'input_blob_name': preprocessed_image}
outputs = sess.run(None, inputs)
# 解析推理结果
output_data = outputs[0]
predicted_label = np.argmax(output_data)
print("Predicted Label:", predicted_label)
4、TensorRT
TensorRT (Tensor Runtime) is a high-performance optimization and deployment framework for deep learning model reasoning developed by NVIDIA. It can accelerate the inference speed of deep learning models and provide low-latency real-time inference on a variety of hardware platforms.
The following is a detailed introduction to the TensorRT framework:
Model optimization and network layer fusion: TensorRT uses a series of optimization techniques to optimize deep learning models, such as accuracy calibration, network layer fusion, tensor rearrangement, etc. These optimization techniques reduce model computation, memory footprint, and latency while maintaining model accuracy.
Tensor core: TensorRT uses tensor core (Tensor Core) to accelerate the calculation of deep neural network. Tensor cores are hardware units in NVIDIA GPUs that can perform highly parallel matrix operations, greatly speeding up model inference.
Automatic mixed precision: TensorRT supports automatic mixed precision, which automatically converts floating-point operations to lower-precision operations to improve inference performance. By leveraging half-precision (FP16) computation, memory bandwidth and model computation can be significantly reduced, resulting in faster inference.
Dynamic shape support: TensorRT supports dynamic shapes (Dynamic Shapes), that is, the calculation graph of the model is dynamically adjusted according to the input shape at runtime. This is very useful for dealing with variable-length sequence data and batch size is not fixed, improving the flexibility and generality of the model.
Cross-platform compatibility: TensorRT provides support for multiple hardware platforms and operating systems, including NVIDIA GPUs, NVIDIA Jetson series embedded devices, and x86-based CPUs. It also integrates with common deep learning frameworks (such as TensorFlow, PyTorch, etc.), so that users can easily convert the trained model into a TensorRT inference engine.
Graph optimization and layer fusion: TensorRT uses graph optimization techniques to reorganize and streamline the computational graph of neural networks and perform layer fusion to reduce memory usage and computation. This optimization can greatly improve inference performance and reduce the storage space of the model.
Support for multiple inference engines: TensorRT provides C++ and Python APIs that users can use to integrate and deploy TensorRT-optimized models. In addition, TensorRT also provides some auxiliary tools, such as the ONNX-TensorRT converter, which is used to convert the model in ONNX format into a TensorRT inference engine.
To sum up, TensorRT is a powerful deep learning model inference deployment framework with features such as model optimization and network layer fusion, tensor core acceleration, automatic mixed precision, dynamic shape support, cross-platform compatibility, graph optimization, and layer fusion. It can significantly improve the inference performance of deep learning models and enable efficient real-time inference on a variety of hardware devices.
Also take handwritten digit recognition as an example, and give the example operation code:
1、准备模型：
首先，需要准备一个训练好的手写数字识别模型，并将其保存为TensorFlow SavedModel或ONNX格式。
2、安装TensorRT：
在开始之前，请确保已经按照官方文档中的指引安装了TensorRT，并设置了必要的环境变量。
3、转换模型：
使用TensorRT提供的Model Optimizer工具将TensorFlow模型或ONNX模型转换为TensorRT格式。打开终端并执行以下命令：
$ trtexec --onnx=/path/to/model/handwritten_digit.onnx --saveEngine=/path/to/model/handwritten_digit.trt
其中，/path/to/model/handwritten_digit.onnx是原始模型的路径，/path/to/model/handwritten_digit.trt是转换后TensorRT模型的保存路径。
4、推理代码： 使用TensorRT的Python API编写一个进行推理的脚本。以下是一个简单的示例代码：
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np
# 加载TensorRT Engine
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
with open('/path/to/model/handwritten_digit.trt', 'rb') as f, trt.Runtime(TRT_LOGGER) as runtime:
engine = runtime.deserialize_cuda_engine(f.read())
# 创建执行上下文
with engine.create_execution_context() as context:
# 分配输入和输出内存
inputs, outputs, bindings, stream = allocate_buffers(engine)
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 对输入数据进行预处理
preprocessed_image = preprocess(image)
# 将输入数据复制到GPU内存中
np.copyto(inputs[0].host, preprocessed_image.ravel())
# 执行推理
context.execute_async(bindings=bindings, stream_handle=stream.handle)
cuda.streams.synchronize()
# 解析推理结果
output_data = np.array(outputs[0].host)
predicted_label = np.argmax(output_data)
print("Predicted Label:", predicted_label)
5、TFLite
TFLite (TensorFlow Lite) is a framework developed by Google for deep learning model inference on mobile devices, embedded systems, and IoT devices. It is designed for resource-constrained devices and is lightweight, fast and efficient.
The following is a detailed introduction to the TFLite framework:
Lightweight Models: TFLite supports lightweight transformation of deep learning models to fit the computing power and storage constraints of mobile devices and embedded systems. Through techniques such as model quantization, pruning, and optimization, TFLite can greatly reduce the size of the model, thereby reducing memory usage and latency.
Fast inference engine: TFLite provides a fast inference engine for mobile devices and embedded systems. These engines can achieve high-performance deep learning model inference by using hardware accelerators (such as GPU, DSP, NPU, etc.) and optimized algorithms.
Support a variety of hardware platforms: TFLite supports a variety of mainstream hardware platforms, including Android and iOS devices, embedded systems such as Raspberry Pi, and some IoT devices. This allows developers to deploy trained and optimized deep learning models to various devices.
Multi-platform compatibility: TFLite can be seamlessly integrated with the TensorFlow framework, allowing users to train and debug models in TensorFlow, and convert them to TFLite format for deployment. In addition, TFLite also provides APIs in multiple programming languages such as C++, Python, and Java, so that developers can use familiar tools and environments.
Dynamic and static graph support: TFLite supports both dynamic and static graph models. For dynamic graph models (such as Keras), TFLite provides TF 2.x TFLite Converter to realize model conversion; for static graph models (such as TensorFlow SavedModel), TFLite provides TF 1.x TFLite Converter to realize model conversion.
Offline and online reasoning: TFLite supports both offline reasoning and online reasoning. Offline reasoning is suitable for scenarios where inference tasks need to be performed locally on the device, while online reasoning is suitable for scenarios that need to interact with cloud services. TFLite provides corresponding APIs and tools to meet different needs.
Extended functions: TFLite also provides some extended functions, such as pose estimation, target detection, speech recognition, etc. These features can help developers build deep learning applications more quickly and provide high-quality prediction results.
To sum up, TFLite is a framework for deep learning model inference on mobile devices, embedded systems, and IoT devices. It has the characteristics of lightweight model, fast inference engine, multiple hardware platform support, multi-platform compatibility, dynamic and static graph support, offline and online inference, and extended functions. TFLite can help developers achieve efficient and fast inference of deep learning models on resource-constrained devices.
Also take handwritten digit recognition as an example, and give an example operation code:
1、准备模型：
首先，需要准备一个训练好的手写数字识别模型，并将其保存为TensorFlow SavedModel或Keras H5格式。
2、安装TFLite库：
在开始之前，请确保已经安装了TFLite库。可以使用以下命令安装：
pip install tensorflow
3、转换模型： 使用TFLite Converter工具将TensorFlow SavedModel或Keras H5模型转换为TFLite格式。打开终端并执行以下命令：
tflite_convert --saved_model_dir=/path/to/model/saved_model/ --output_file=/path/to/model/handwritten_digit.tflite
或：
tflite_convert --keras_model_file=/path/to/model/h5_model.h5 --output_file=/path/to/model/handwritten_digit.tflite
4、推理代码： 使用TFLite的Python API编写一个进行推理的脚本。以下是一个简单的示例代码：
import tensorflow as tf
import numpy as np
# 加载并初始化模型
interpreter = tf.lite.Interpreter(model_path='/path/to/model/handwritten_digit.tflite')
interpreter.allocate_tensors()
# 获取输入和输出张量索引
input_index = interpreter.get_input_details()[0]['index']
output_index = interpreter.get_output_details()[0]['index']
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 对输入数据进行预处理
preprocessed_image = preprocess(image)
# 设置输入张量的值
interpreter.set_tensor(input_index, preprocessed_image)
# 执行推理
interpreter.invoke()
# 获取输出张量的值
output_data = interpreter.get_tensor(output_index)
predicted_label = np.argmax(output_data)
print("Predicted Label:", predicted_label)
6、TorchServe
TorchServe (Torch Server) is a framework developed by the PyTorch team for inference deployment of deep learning models. It aims to provide a simple and scalable model deployment solution, enabling users to quickly deploy trained PyTorch models to production environments.
The following is a detailed introduction of the TorchServe framework:
Model deployment and management: TorchServe provides an easy and powerful way to deploy and manage deep learning models. Users can easily start, stop and manage multiple model instances using the TorchServe command line tool. In addition, TorchServe also supports model hot update, that is, updating deployed models without service interruption.
Multi-model support: TorchServe supports deploying multiple models at the same time and provides access to these models through RESTful API. This enables users to run multiple model instances concurrently on the same server and perform model inference through API calls.
High performance and low latency: TorchServe achieves high performance and low latency model inference by using technologies such as multi-threading and asynchronous operations. It also supports features such as model batching, model warming, and request caching to further improve inference performance.
Flexible model configuration: TorchServe uses a simple JSON configuration file to define the inference behavior of the model. This allows users to easily configure the model's input and output formats, pre-processing and post-processing operations, etc., and adjust them according to their needs.
Custom reasoning logic: TorchServe allows users to extend the functionality of the framework by writing custom reasoning logic. Users can write custom inference handlers to perform additional operations, such as data transformations, logging, etc., before or after inference.
Cross-platform compatibility: TorchServe can run on multiple platforms, including Linux, Windows, and macOS, etc. It supports CPU and GPU inference and seamlessly integrates with common deep learning libraries such as PyTorch and TorchScript.
Community support and activity: TorchServe is an open source project-based framework with an active developer community providing extensive documentation, examples, and tutorials. This allows users to get help and support from community resources and participate in improving and contributing to the framework.
To sum up, TorchServe is a framework for inference deployment of deep learning models, featuring model deployment and management, multi-model support, high performance and low latency, flexible model configuration, custom inference logic, cross-platform compatibility, and active community support. TorchServe provides a simple and scalable way to deploy the trained PyTorch model to the production environment and provide users with efficient and reliable model inference services.
Also take handwritten numbers as an example, and give the example operation code:
1、准备模型：
首先，需要准备一个训练好的手写数字识别模型，并将其保存为PyTorch模型文件（通常是.pth或.pt格式）。
2、安装TorchServe和TorchVision：
在开始之前，请确保已经安装了TorchServe和TorchVision库。可以使用以下命令安装：
pip install torchserve torch torchvision
3、转换为TorchScript格式： 使用PyTorch提供的torch.jit.trace函数将模型转换为TorchScript格式。以下是一个简单的示例代码：
import torch
import torchvision.transforms as transforms
# 加载并初始化模型
model = torch.load('/path/to/model/handwritten_digit.pth')
model.eval()
# 创建示例输入
example_input = torch.rand(1, 1, 28, 28) # 替换成实际的手写数字图片数据
# 将模型转换为TorchScript格式
traced_model = torch.jit.trace(model, example_input)
traced_model.save('/path/to/model/handwritten_digit.pt')
4、配置和启动TorchServe： 创建一个配置文件 config.properties，其中包含了模型的相关信息、预处理和后处理操作等。以下是一个示例配置文件的内容：
model_name=handwritten_digit
model_file=/path/to/model/handwritten_digit.pt
handler=app.HandwrittenDigitHandler
batch_size=1
max_batch_delay=5000
initial_workers=1
synchronous=false
在上述配置文件中，model_name 是模型名称，model_file 是TorchScript模型的路径，handler 是自定义的请求处理类，batch_size 是批处理大小，max_batch_delay 是设置延迟时间，initial_workers 是初始工作进程数，synchronous 设置是否同步执行。
启动TorchServe服务器并加载模型：
torchserve --start --model-store=/path/to/model/store --models handwritten_digit=/path/to/config.properties
其中，/path/to/model/store 是模型存储目录的路径。
5、发送推理请求： 使用HTTP POST请求发送推理请求到TorchServe服务器。以下是一个简单的示例代码：
import requests
import numpy as np
# 准备要推理的手写数字图片数据
image = np.random.rand(28, 28) # 替换成实际的手写数字图片数据
# 对输入数据进行预处理
preprocessed_image = preprocess(image)
# 构建推理请求的URL和数据
url = 'http://localhost:8080/predictions/handwritten_digit'
data = {'input': preprocessed_image.tolist()}
# 发送推理请求
response = requests.post(url, json=data)
# 解析推理结果
output_data = response.json()
predicted_label = np.argmax(output_data['predictions'])
print("Predicted Label:", predicted_label)