There is no "middleman earning the difference", OpenVINO™ directly supports PyTorch model objects

Author: Yang Yicheng

1. Background

As one of the most popular open source deep learning frameworks, PyTorch's ease of use and flexibility have made it popular among academic and research communities. Previously, OpenVINO™'s support for the PyTorch model was only in the ONNX transitional stage. It was necessary to export the PyTorch dynamic model to the ONNX static format before it could be directly loaded offline by the OpenVINO™ runtime, although PyTorch also provided the official torch.onnx. The export interface helps developers to export ONNX models, but there is such a "middleman" there after all, and a lot of extra configuration work also brings inconvenience to OpenVINO™ developers, such as dynamic/static input settings, and opset version settings wait.

2. OpenVINO™ directly supports PyTorch model objects

[RL1]  [RL2]  [YE3]  [LR4]  [LR5]  [LR6]  [YE7] 

With the release of OpenVINO™ 2023.0, a new PyTorch front-end is pre-installed in the OpenVINO™ tool library, providing developers with a new PyTorch model support path and bringing a more friendly user experience - OpenVINO™' s mo tool It is possible to directly convert PyTorch model objects to OpenVINO™ model objects, and developers do not need to use the ONNX model as an intermediate transition.

import torchvision

import torch

from openvino.tools.mo import convert_model



model = torchvision.models.resnet50(pretrained=True)

ov_model = convert_model(model)

Compared with ONNX as an intermediate transition method, the new PyTorch front end has the following characteristics:

	ONNX Transition	New PyTorch frontend
offline conversion	support	support
online conversion	Not supported, need to export ONNX file	support
conversion steps	cumbersome	Only need to call the interface once
parameter configuration	many	medium
Operator support	Rich	medium

Currently supported PyTorch model objects are:

• torch.nn.Module
• torch.jit.ScriptModule
• torch.jit.ScriptFunction

Inside OpenVINO™, the PyTorch front-end is based on TorchScript for model export, and TorchScript supports two model export modes, one is called Tracing and the other is called Scripting. Among them, Tracing refers to PyTorch's tracking of the module operators that run through when the model is running, real-time construction of a calculation flow graph, and finally summarizing it into an intermediate representation. Trace is a double-edged sword. The advantage is that users do not need to understand the details of Python code , whether it is Function, Module, Generators, or Coroutines, Tracing will faithfully record the passed Tensor and Tensor Function, which is very suitable for simple modules and functions that do not involve data-related control flow, such as standard convolutional neural networks. The disadvantage is that Tracing Unaware of control flow and dynamics of computation graphs, such as if statements or loops. For example, he will expand the loop. On the one hand, it may increase the space for compilation optimization. On the other hand, if the loop is dynamically lengthened when different infers are used, then Tracing will not be able to perceive this, and will only record the loop during Tracing. down. To transform modules and functions that contain data-dependent control flow, a Scripting mechanism is provided that parses from the Python source code level, rather than being built at runtime. Scripting will understand all the code, and really perform syntax analysis and other operations like a compiler. Scripting is equivalent to a DSL embedded in Python/Pytorch, and its grammar is only a subset of PyTorch grammar, which means that there are some ops and grammars that Scripting does not support, so you will encounter problems when compiling.

In the example just now, the PyTorch front-end uses Scripting to export the model. If you want to use the Tracing method, you can add an example_input parameter to the interface. At this time, the PyTorch front-end will call the Tracing method first. When the Tracing method fails, call Scripting mode.

import torchvision
import torch
from openvino.tools.mo import convert_model

model = torchvision.models.resnet50(pretrained=True)
ov_model = convert_model(model, example_input=torch.zeros(1, 3, 100, 100))

The data formats currently supported by examle_input are:

• openvino.runtime.Tensor
• torch.Tensor
• np.ndarray
• list or tuple with tensors (openvino.runtime.Tensor / torch.Tensor / np.ndarray)
• dictionary where key is the input name, value is the tensor (openvino.runtime.Tensor / torch.Tensor / np.ndarray)

It is worth noting that the above two examples export dynamic input model objects. If you want to specify the input shape of the model, you can add an additional parameter input_shape/input again, pass the input shape as a parameter, and choose one. The case can refer to the actual combat part below.

Finally, if developers want to export static IR files for subsequent use, they can also call the following interface to serialize OpenVINO™ model objects:

serialize(ov_model, str(ir_model_xml))

Three, BERT model case combat

Next, let's use an example to see how to complete the whole process from BERT model conversion to quantification.

1. Get the PyTorch model object

torch_model = BertForSequenceClassification.from_pretrained(PRETRAINED_MODEL_DIR)

2. Set the model parameters and convert it into an OpenVINO™ model object. Since BERT is a multi-input model, an additional input=input_info parameter is added here, which can be used to specify the shape and data type of each input in the multi-input model.

input_shape = PartialShape([1, -1])
input_info = [("input_ids", input_shape, np.int64),("attention_mask", input_shape, np.int64),("token_type_ids", input_shape, np.int64)]
default_input = torch.ones(1, MAX_SEQ_LENGTH, dtype=torch.int64)
inputs = {
    "input_ids": default_input,
    "attention_mask": default_input,
    "token_type_ids": default_input,
}
model = convert_model(torch_model, example_input=inputs, input=input_info)

3. Prepare the verification data set and start the quantization. The model obtained in the previous step is of openvino.runtime.Model type, which can be directly loaded by the NNCF tool

calibration_dataset = nncf.Dataset(data_source, transform_fn)
# Quantize the model. By specifying model_type, we specify additional transformer patterns in the model.
quantized_model = nncf.quantize(model, calibration_dataset,
                                model_type=ModelType.TRANSFORMER)

4. Compile the quantized model object and perform inference

compiled_quantized_model = core.compile_model(model=quantized_model, device_name="CPU")
output_layer = compiled_quantized_model.outputs[0]
result = compiled_quantized_model(inputs)[output_layer]
result = np.argmax(result)
print(f"Text 1: {sample['sentence1']}")
print(f"Text 2: {sample['sentence2']}")
print(f"The same meaning: {'yes' if result == 1 else 'no'}")

The final result is as follows:

Text 1: Wal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .

Text 2: It has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .

The same meaning: yes

For complete examples and performance accuracy comparison, please refer to:

openvino_notebooks/notebooks/105-language-quantize-bert/105-language-quantize-bert.ipynb at main · openvinotoolkit/openvino_notebooks · GitHub

Four. Summary

As the latest version recently released, the mo tool in OpenVINO™ 2023.0 can directly convert PyTorch model objects into OpenVINO™ objects without intermediate transition through ONNX, eliminating the need for offline conversion and additional configuration for developers, bringing Come to a more friendly user experience. Since this function is in a pre-release state, some operators may not support it. At this time, developers can still use the previous path to convert PyTorch models relying on the ONNX front-end.

---

 [RL1]We should list out the ‘benefits’ of why no conversion is great. I mean to us conversion or not doesn’t really matter. List out means 1. 2. 3. 4. 5. like value prop.

 [RL2]Also, we shall list out the benefits still with ONNX? If we can, so that way this post isn’t as biased about ONNX is bad, and some people may take this wrongly

 [YE3] Done

We should not "CROSS OUT" ONNX. It also looks bad on ONNX when we do so to other brand. [LR4]

Best is to just show PyTorch and OpenVINO together story only. [LR5]

 [LR6]and you can leave ONNX on the side and make it look lonely, that’s ok :)

 [YE7]Done