Image classification based on PyTorch's C++API running model

Introduction to TorchScript

TorchScript is an intermediate form of the PyTorch model, which can be run in a high-performance environment (such as C++).

A simple example is as follows:

import torch
#import torchvision

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()

    def forward(self, x, h):
        new_h = torch.tanh(x + h)
        return new_h, new_h

my_cell = MyCell()
x = torch.rand(3, 4)
h = torch.rand(3, 4)
print(my_cell(x, h))

 Output result:

        Based on the above example, we torch.nn.Modulecreated a class MyCelland defined a constructor. The constructor here only calls the superfunction.
super()Function is a method used to call the parent class (super class). superIt is used to solve the problem of multiple inheritance. It is no problem to call the parent class method directly with the class name when using single inheritance, but if you use multiple inheritance, it will involve various issues such as search order and repeated calls. At the same time, we also define a forwardfunction. The forwardfunction input here is 2 parameters and returns 2 results. The actual content of the forward function is not very important, but it is a pseudo RNN unit, that is, the real scene of the function is applied to the loop.

 

We further modify the above MyCellclass, add a self.linearmember attribute (a function) on the original basis , and forwardcall the member in the function. torch.nn.LinearIt is a standard module in PyTorch, which completes the nested combination of modules.

import torch


class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)
 
    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h
 
my_cell = MyCell()
x = torch.rand(3, 4)
h = torch.rand(3, 4)
print(my_cell)
print(my_cell(x, h))

 Output result:

        When printing a module, the output is the subclass hierarchy of the module. For example, mycellthe result of the above printing is the linearsubclass and its parameters. By combining modules in this way, you can easily create models with reusable components.
In addition, it can be seen from the output results that there are grad_fn. This is the information given by PyTorch's automatic differentiation and derivation, called autograd. In short, the system allows us to calculate derivatives through potentially complex procedures. This design provides great flexibility for model creation.

        Below we use examples to further illustrate the flexibility of model construction. Added on the basis of the above MyDecisionGate, the control flow in the form of loop or if statement is used in this module.

import torch

class MyDecisionGate(torch.nn.Module):
  def forward(self, x):
    if x.sum() > 0:
      return x
    else:
      return -x

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.dg = MyDecisionGate()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.dg(self.linear(x)) + h)
        return new_h, new_h

my_cell = MyCell()
x = torch.rand(3, 4)
h = torch.rand(3, 4)
print(my_cell)
print(my_cell(x, h))

Output result:

Tracing

In short, given the flexible and dynamic nature of native PyTorch, TorchScript also provides tools to capture model definitions. One of the core concepts is 模型追踪(tracing).

import torch

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.trace(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

Output result: 

                    

        As before, instantiate MyCell, but this time, use the torch.jit.tracemethod to call Module, and then pass in the sample input of the network. What exactly is this for? It has called Module, recorded the operations that occurred while Module was running, and created an torch.jit.ScriptModuleinstance (an instance of TracedModule). TorchScript records its definition in an intermediate representation (or IR), which is usually called a graph in deep learning. We can .graphview the graph by accessing the properties: 

print(traced_cell.graph)

Output result: 

However, this is a very low-level representation, and most of the information contained in the graph is not useful to the end user. Instead, we can use .codeattributes to provide code with an explanation of Python syntax: 

print(traced_cell.code)

Output result: 

 

So why do we do all this? There are several reasons:

1. TorchScript code can be called in its own interpreter, which is basically a restricted Python interpreter . The interpreter does not acquire the global interpreter lock , so many requests can be processed simultaneously on the same instance.
2. This format allows us to save the entire model to disk and load it in another environment, such as in a service written in a language other than Python.
3. TorchScript provides us with a representation, through TorchScript we can perform compiler optimization on the code to provide more efficient execution.
4. TorchScript can interface with many backend/device runtimes, which require a broader view of the program than a single operation.

You can see traced_cellthat the result of the call is the same as the result of directly executing the Python module:
Run:

import torch

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.trace(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)
print(my_cell(x, h))
print(traced_cell(x, h))

Output result:

Use Scripting to Convert Modules

      We use the second version of the module for traced_cell(x, h)a reason, instead of using a version of a submodule with control flow. Let us illustrate the reason behind it with the following example.

import torch

class MyDecisionGate(torch.nn.Module):
  def forward(self, x):
    if x.sum() > 0:
      return x
    else:
      return -x

class MyCell(torch.nn.Module):
    def __init__(self, dg):
        super(MyCell, self).__init__()
        self.dg = dg
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.dg(self.linear(x)) + h)
        return new_h, new_h

my_cell = MyCell(MyDecisionGate())
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.trace(my_cell, (x, h))
print(traced_cell.code)

 Output result:

According to .codethe output, there is if-elseno trace of the branch that can be found ! why? TracingDo exactly what we say: run the code, record what happens, and construct a one that can do this ScriptModule. Unfortunately, during this operation, information such as control flow is erased.
So how TorchScriptto truthfully represent this module in? PyTorch provides a script compiler , which can directly divide the Python source code to convert it into TorchScript. MyDecisionGateConvert the above-mentioned using script compiler: 

scripted_gate = torch.jit.script(MyDecisionGate())  # 看这里

my_cell = MyCell(scripted_gate)
traced_cell = torch.jit.script(my_cell)  # 看这里
print(traced_cell.code)
这部分我运行出了些问题，可能是软件版本太低了，有时间使用PyTorch版本1.2.0+cu92试一试，先copy一下，有时间再查原因吧。。。。。

operation result:

def forward(self,
    x: Tensor,
    h: Tensor) -> Tuple[Tensor, Tensor]:
  _0 = self.linear
  _1 = _0.weight
  _2 = _0.bias
  if torch.eq(torch.dim(x), 2):
    _3 = torch.__isnot__(_2, None)
  else:
    _3 = False
  if _3:
    bias = ops.prim.unchecked_unwrap_optional(_2)
    ret = torch.addmm(bias, x, torch.t(_1), beta=1, alpha=1)
  else:
    output = torch.matmul(x, torch.t(_1))
    if torch.__isnot__(_2, None):
      bias0 = ops.prim.unchecked_unwrap_optional(_2)
      output0 = torch.add_(output, bias0, alpha=1)
    else:
      output0 = output
    ret = output0
  _4 = torch.gt(torch.sum(ret, dtype=None), 0)
  if bool(_4):
    _5 = ret
  else:
    _5 = torch.neg(ret)
  new_h = torch.tanh(torch.add(_5, h, alpha=1))
  return (new_h, new_h)

Now, we can faithfully capture the behavior of the program in TorchScript. Now try to run the program:

# New inputs
x, h = torch.rand(3, 4), torch.rand(3, 4)
print(traced_cell(x, h))

operation result: 

(tensor([[ 0.3430, -0.3471,  0.7990,  0.8313],
        [-0.4042, -0.3058,  0.7758,  0.8332],
        [-0.3002, -0.3926,  0.8468,  0.7715]],
       grad_fn=<DifferentiableGraphBackward>), tensor([[ 0.3430, -0.3471,  0.7990,  0.8313],
        [-0.4042, -0.3058,  0.7758,  0.8332],
        [-0.3002, -0.3926,  0.8468,  0.7715]],
       grad_fn=<DifferentiableGraphBackward>))

Note that the PyTorch version of this experiment is the 1.1.0+cu9.0。建议使用PyTorch version1.2.0+cu92。

Mixed scripting and tracing

To be added. . . . . . . . . . . . .

Load TorchScript model in C++

Step 1: Convert the PyTorch model to Torch Script

The PyTorch model needs to be Torch Scriptimplemented from Python to C++ . Torch Script is a representation of the PyTorch model, which can be understood, compiled and serialized by the Torch Script compiler. If you write a PyTorch model with the ordinary "eager" API, you must first convert the model to Torch Script.

The previous chapters have introduced 2 ways to convert PyTorch model to Torch Script. The first is tracing, which evaluates the structure of the model through instance inputs and records the flow of these inputs through the model. This method is suitable for situations where the model has limited use of control flow. The second method is to add explicit comments to the model so that the Torch Script compiler can directly parse and compile the model code. For more detailed information, please refer to Torch Script reference

By Tracing

To convert a PyTorch model to Torch Script by tracking, the model instance with sample input must be input to the torch.jit.tracefunction. This will produce an torch.jit.ScriptModuleobject that embeds the tracking of the model evaluation in the forward method.
Specific usage examples are as follows:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

The tracked ScriptModuleobject can now be regarded as a regular PyTorch module.

output = traced_script_module(torch.ones(1, 3, 224, 224))
print(output[0, :5])

Output result:

tensor([0.7741, 0.0539, 0.6656, 0.7301, 0.2207], grad_fn=<SliceBackward>)

Via Annotation

In some cases, for example, if the model adopts a specific form of control flow, it may be a better choice to write the model directly in Torch Script and label the model accordingly. Take the following Pytorch model as an example:

import torch

class MyModule(torch.nn.Module):
    def __init__(self, N, M):
        super(MyModule, self).__init__()
        self.weight = torch.nn.Parameter(torch.rand(N, M))

    def forward(self, input):
        if input.sum() > 0:
          output = self.weight.mv(input)
        else:
          output = self.weight + input
        return output

因为这个模块中的forward方法使用依赖于输入的控制流依，这种模块不适合于追踪方法。相反，可以将其转换为ScriptModule。为了将模块转换为ScriptModule，需要用torch.jit.script编译模块：

class MyModule(torch.nn.Module):
    def __init__(self, N, M):
        super(MyModule, self).__init__()
        self.weight = torch.nn.Parameter(torch.rand(N, M))

    def forward(self, input):
        if input.sum() > 0:
          output = self.weight.mv(input)
        else:
          output = self.weight + input
        return output

my_module = MyModule(10,20)
sm = torch.jit.script(my_module)

In addition, for nn.Modulemethods that are not needed in (because TorchScript currently does not support some python features), you can use @torch.jit.ignorethem to remove them.

Step 2: Serialize the Script Module to the file

For the obtained ScriptModuleobject (whether it is obtained by the tracing method or the annotation method), it can be serialized into a file for subsequent use in other environments (such as C++). The specific serialization method is as follows:

traced_script_module.save("traced_resnet_model.pt")

• If you want to serialize the module at the same time my_module, you can use it my_module.save("my_module_model.pt").

Step 3: Load the Torch Script module in C++

Loading the serialized PyTorch model in C++ requires the PyTorch C++ API, which is a LibTorchlibrary. LibTorchThere are shared libraries, header files and CMake build configuration files.

The most simplified C++ application

example-app.cppThe contents are as follows:

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }


  torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error& e) {
    std::cerr << "error loading the model\n";
    return -1;
  }

  std::cout << "ok\n";
}

The header file <torch/script.h>includes all the dependencies in the LibTorch library necessary to run the example. The above example receives the serialized ScriptModulefile and torch::jit::load()loads the serialized file, and the result is an torch::jit::script::Moduleobject.

Build dependencies and create

The content of CMakeLists.txt corresponding to the above code is as follows:

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)

find_package(Torch REQUIRED)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 11)

Download libtorch from the official and unzip it:

The libdirectory contains the shared libraries required for linking; includecontains the header files used in the program; the sharedirectory contains the necessary CMake configuration to facilitate find_package(Torch)the use of the above commands.

Finally, you need to build the application. Suppose the directory layout is as follows:

example-app/
  CMakeLists.txt
  example-app.cpp

You can run the following command to example-app/build the application from the folder:

mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=/home/data1/devtools/libtorch/ ..
make

This DCMAKE_PREFIX_PATHvalue is libtorchthe location to unpack after downloading .
After compilation, the operation mode is as follows:

./example-app <path_to_model>/traced_resnet_model.pt

Step 4: Execute Script Module in C++ 

The above introduction has been able to load serialized ResNet18 in C++, and now what needs to be done is to run the model for reasoning. details as follows:

// Create a vector of inputs.
std::vector<torch::jit::IValue> inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));

// Execute the model and turn its output into a tensor.
at::Tensor output = module.forward(inputs).toTensor();
std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';

The first 2 lines of the above code are the input of the model, and then call script::Modulethe forwardmethod in, the type of the returned result is IValuethat it needs to be further toTensor()converted to tensor.

Note: If you want to run the model with GPU, you only need to deal with the model as follows: model.to(at::kCUDA);. At the same time, it is necessary to ensure that the input of the model is also in CUDA memory, which can be implemented in the following ways:, tensor.to(at::kCUDA)a new tensor located in CUDA memory will be returned.

Image classification example

Environmental preparation

You need to install cmake, opencv, PyTroch 1.2 in advance. During the opencv installation process, there may be some environmental installation problems such as the gcc version (gcc5.2 used in this article) is too low, and I will explain it here.

Load model in C++

Take the resnet18 model for image classification as an example.

Step 1: Convert the PyTorch model to Torch Script

Run the following script:

import torch
import torchvision
from torchvision import transforms
from PIL import Image
from time import time
import numpy as np

# An instance of your model.
model = torchvision.models.resnet18(pretrained=True)
model.eval()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

# evalute time
batch = torch.rand(64, 3, 224, 224)
start = time()
output = traced_script_module(batch)
stop = time()
print(str(stop-start) + "s")

# read image
image = Image.open('dog.png').convert('RGB')
default_transform = transforms.Compose([
        transforms.Resize([224, 224]),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
      ])
image = default_transform(image)

# forward
output = traced_script_module(image.unsqueeze(0))
print(output[0, :10])

# print top-5 predicted labels
labels = np.loadtxt('synset_words.txt', dtype=str, delimiter='\n')

data_out = output[0].data.numpy()
sorted_idxs = np.argsort(-data_out)

for i,idx in enumerate(sorted_idxs[:5]):
  print('top-%d label: %s, score: %f' % (i, labels[idx], data_out[idx]))

Get model.pt

Step 2: Call Torch Script in C++

(1) You need to download LibTorchand unpack first, and you need to specify the path of the lib when compiling make.
(2) Use the cmake tool to compile the business code, that is, the code using Torch Script

mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=/home/aaron/WORK/tb1/libtorch/ ..
make

operation result:

Attach the complete code: 

#include "torch/script.h"
#include "torch/torch.h" 
//#include "torch/Tensor.h" 
#include "opencv2/opencv.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/imgproc/types_c.h"

#include <iostream>
#include <memory>
#include <string>
#include <vector>

/* main */
int main(int argc, const char* argv[]) {
  if (argc < 4) {
    std::cerr << "usage: example-app <path-to-exported-script-module> "
      << "<path-to-image>  <path-to-category-text>\n";
    return -1;
  }

  // Deserialize the ScriptModule from a file using torch::jit::load().
  //std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);
  torch::jit::script::Module module = torch::jit::load(argv[1]);

  //assert(module != nullptr);
  std::cout << "load model ok\n";

  // Create a vector of inputs.
  std::vector<torch::jit::IValue> inputs;
  inputs.push_back(torch::rand({64, 3, 224, 224}));

  // evalute time
  double t = (double)cv::getTickCount();
  module.forward(inputs).toTensor();
  t = (double)cv::getTickCount() - t;
  printf("execution time = %gs\n", t / cv::getTickFrequency());
  inputs.pop_back();

  // load image with opencv and transform
  cv::Mat image;
  image = cv::imread(argv[2], 1);
  cv::cvtColor(image, image, CV_BGR2RGB);
  cv::Mat img_float;
  image.convertTo(img_float, CV_32F, 1.0/255);
  cv::resize(img_float, img_float, cv::Size(224, 224));
  //std::cout << img_float.at<cv::Vec3f>(56,34)[1] << std::endl;
  //auto img_tensor = torch::CPU(torch::kFloat32).tensorFromBlob(img_float.data, {1, 224, 224, 3});

  auto img_tensor = torch::from_blob(img_float.data, {1, 224, 224, 3}); //.permute({0, 3, 1, 2}).to(torch::kCUDA);
 
  img_tensor = img_tensor.permute({0,3,1,2}); //.to(torch::kCUDA);

  img_tensor[0][0] = img_tensor[0][0].sub_(0.485).div_(0.229);
 
  img_tensor[0][1] = img_tensor[0][1].sub_(0.456).div_(0.224);
  img_tensor[0][2] = img_tensor[0][2].sub_(0.406).div_(0.225);

  //auto img_var = torch::autograd::make_variable(img_tensor, false);
  //torch::Tensor img_var = torch::autograd::make_variable(img_tensor, false);
 
  inputs.push_back(img_tensor);
  
  // Execute the model and turn its output into a tensor.
  torch::Tensor out_tensor = module.forward(inputs).toTensor();
 
  std::cout << out_tensor.slice(/*dim=*/1, /*start=*/0, /*end=*/10) << '\n';


  // Load labels
  std::string label_file = argv[3];
  std::ifstream rf(label_file.c_str());
  CHECK(rf) << "Unable to open labels file " << label_file;
  std::string line;
  std::vector<std::string> labels;
  while (std::getline(rf, line))
    labels.push_back(line);

  // print predicted top-5 labels
  std::tuple<torch::Tensor,torch::Tensor> result = out_tensor.sort(-1, true);
  torch::Tensor top_scores = std::get<0>(result)[0];
  torch::Tensor top_idxs = std::get<1>(result)[0].toType(torch::kInt32);
  
  auto top_scores_a = top_scores.accessor<float,1>();
  auto top_idxs_a = top_idxs.accessor<int,1>();

  for (int i = 0; i < 5; ++i) {
    int idx = top_idxs_a[i];
    std::cout << "top-" << i+1 << " label: ";
    std::cout << labels[idx] << ", score: " << top_scores_a[i] << std::endl;
  }

  return 0;
}

Reference

https://pytorch.org/blog/model-serving-in-pyorch/
https://medium.com/datadriveninvestor/deploy-your-pytorch-model-to-production-f69460192217
https://github.com/iamhankai/cpp-pytorch
https://pytorch.org/tutorials/advanced/cpp_export.html#step-1-converting-your-pytorch-model-to-torch-script

https://blog.csdn.net/jonado13/article/details/108280029

https://blog.csdn.net/ljp1919/article/details/102514357?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.control