1. Description

Beginners text : This text requires an entry-level programming background and a basic understanding of machine learning. Tensors are the primary way of representing data in deep learning algorithms. They are widely used to implement inputs, outputs, parameters, and internal state during algorithm execution.

In this story, we will learn how to use the feature tensor API to develop our C++ algorithm. Specifically, we will discuss:

what is tensor
How to define tensor in C++
How to calculate tensor operations
Tensor reduction and convolution

At the end of this article, we will implement Softmax as an illustrative example of applying tensors to deep learning algorithms.

。

2. What is tensor?

Tensors are grid-like data structures that generalize the concept of vectors and matrices with any number of axes. In machine learning, we usually use the word " dimension " instead of "axis". The number of different dimensions of a tensor is also known as the tensor rank :

different rank tensors

In practice, we use tensors to represent data in algorithms and perform arithmetic operations with them.

The simpler operations we can perform with tensors are so-called element-wise operations: given two operand tensors of the same dimension, this operation produces a new tensor of the same dimension with the value of each coefficient is obtained from binary evaluation of the individual elements in the operands:

coefficient multiplication

The above example is a graphical representation of the product of coefficients of two rank 2 tensors. This operation still works for any two tensors, since they have the same dimensions.

Like matrices, we can use tensors to perform other more complex operations such as matrix-like product, convolution, shrinkage, reduction, and countless geometric operations. In this story, we'll learn how to use the Feature Tensor API to perform some of these tensor operations, focusing on the most important operations for implementing deep learning algorithms.

3. How to declare and use tensors in C++

As we all know, Eigen is a linear algebra library widely used for matrix calculations. In addition to the well-known support for matrices, Eigen also has a (unsupported) tensor module.

While the Eigen Tensor API says it's not supported, it's actually well supported by the developers of Google's TensorFlow framework.

We can easily define tensors using traits:

#include <iostream>

#include <unsupported/Eigen/CXX11/Tensor>

int main(int, char **)
{

    Eigen::Tensor<int, 3> my_tensor(2, 3, 4);
    my_tensor.setConstant(42);

    std::cout << "my_tensor:\n\n" 
              << my_tensor << "\n\n";

    std::cout << "tensor size is " << my_tensor.size() << "\n\n"; 

    return 0;
}

The bank

Eigen::Tensor<int, 3> my_tensor(2, 3, 4);

Create a Tensor object and allocate the memory required to store the integers. In this example, is a rank 3 tensor where the first dimension has size 2, the second dimension has size 3, and the last dimension has size 4. We can express it as follows:2x3x4my_tensormy_tensor

We can set tensor data if needed:

my_tensor.setValues({
   
   {
   
   {1, 2, 3, 4}, {5, 6, 7, 8}}});

std::cout << "my_tensor:\n\n" << my_tensor << "\n\n";

Or use random values instead. For example, we can do:

Eigen::Tensor<float, 2> kernel(3, 3);
kernel.setRandom();
std::cout << "kernel:\n\n" << kernel << "\n\n";

And use this kernel later to perform convolution. We'll cover convolutions shortly in this story. First, let's learn how to use TensorMaps.

4. Use Eigen::TensorMap to create a tensor view

Sometimes, we allocate some data and just want to manipulate it using tensors. Similar to but instead of allocating new data, it's just a view of the data passed as a parameter. Check the following examples:Eigen::TensorMapEigen::Tensor

//an vector with size 12
std::vector<float> storage(4*3);

// filling vector from 1 to 12
std::iota(storage.begin(), storage.end(), 1.);

for (float v: storage) std::cout << v << ','; 
std::cout << "\n\n";

// setting a tensor view with 4 rows and 3 columns
Eigen::TensorMap<Eigen::Tensor<float, 2>> my_tensor_view(storage.data(), 4, 3);

std::cout << "my_tensor_view before update:\n\n" << my_tensor_view << "\n\n";

// updating the vector
storage[4] = -1.;

std::cout << "my_tensor_view after update:\n\n" << my_tensor_view << "\n\n";

// updating the tensor
my_tensor_view(2, 1) = -8;

std::cout << "vector after two updates:\n\n";
for (float v: storage) std::cout << v << ','; 
std::cout << "\n\n";

In this example, it is easy to see that (by default) tensors in the Feature Tensor API are col -major . col-major and row-major refer to how grid data is stored in linear containers (check out this article on Wikipedia ):

source

While we can use row-sized tensors, it is not recommended :

Currently only the default column-major layout is fully supported, so trying to use a row-major layout is not recommended at this time.

Eigen::TensorMapVery useful because we can use it to save memory, which is crucial for demanding applications like deep learning algorithms.

5. Perform unary and binary operations

The Eigen Tensor API defines common arithmetic overloaded operators, which make programming with Tensors intuitive and straightforward. For example, we can add and subtract tensors:

Eigen::Tensor<float, 2> A(2, 3), B(2, 3);
A.setRandom();
B.setRandom();

Eigen::Tensor<float, 2> C = 2.f*A + B.exp();

std::cout << "A is\n\n"<< A << "\n\n";
std::cout << "B is\n\n"<< B << "\n\n";
std::cout << "C is\n\n"<< C << "\n\n";

The feature tensor API has several other element-wise functions such as , , and . Additionally, we can use as follows:.exp()sqrt()log()abs()unaryExpr(fun)

auto cosine = [](float v) {return cos(v);};
Eigen::Tensor<float, 2> D = A.unaryExpr(cosine);
std::cout << "D is\n\n"<< D << "\n\n";

Similarly, we can use:binaryExpr

auto fun = [](float a, float b) {return 2.*a + b;};
Eigen::Tensor<float, 2> E = A.binaryExpr(B, fun);
std::cout << "E is\n\n"<< E << "\n\n";

6. Lazy evaluation and the auto keyword

The Google engineers who developed the Eigen Tensor API followed the same strategy as at the top of the Eigen library. One of these strategies, and probably the most important one, is how expressions are evaluated lazily.

The lazy evaluation strategy involves delaying the actual evaluation of an expression so that multiple chained expressions can be combined into a single optimized equivalent expression. So instead of evaluating multiple separate expressions incrementally, optimized code evaluates only one expression, aiming to exploit the resulting overall performance.

For example, the expression does not actually compute the sum of A and B if sum is a tensor. In effect, the expression produces a special object that knows how to evaluate. The actual operation will only be performed when this special object is assigned to an actual tensor. In other words, in the following statement:ABA + BA + BA + B

auto C = A + B;

CNot the actual result, but just a computed object (really an object) that knows how to compute. Only when assigned to a tensor object (object of type , , etc.) will it be evaluated to provide the correct tensor value:A + BEigen::TensorCwiseBinaryOpA + BCEigen::TensorEigen::TensorMapEigen::TensorRef

Eigen::Tensor<...> T = C;
std::cout << "T is " << T << "\n\n";

Of course, this doesn't make sense for small operations like this. However, this behavior is useful for long chains of operations where computation can be optimized before actual evaluation. In resume, as a general guideline, instead of writing code like this:A + B

Eigen::Tensor<...> A = ...;
Eigen::Tensor<...> B = ...;
Eigen::Tensor<...> C = B * 0.5f;
Eigen::Tensor<...> D = A + C;
Eigen::Tensor<...> E = D.sqrt();

We should write code like this:

Eigen::Tensor<...> A = ...;
Eigen::Tensor<...> B = ...;
auto C = B * 0.5f;
auto D = A + C;
Eigen::Tensor<...> E = D.sqrt();

The difference is that in the former, they are actually objects, while in the latter code, they are just lazy computation operations.CDEigen::Tensor

In recovery, it is better to use lazy computation to evaluate long chains of operations, as the chain will be optimized internally, ultimately resulting in faster execution.

7. Geometric operations

Geometric operations produce tensors with varying dimensions and sometimes sizes. Examples of these operations include: , , , and .reshapepadshufflestridebroadcast

It's worth noting that the feature tensor API has no operations. However, we can simulate it using:transposetransposeshuffle

auto transpose(const Eigen::Tensor<float, 2> &tensor) {
    Eigen::array<int, 2> dims({1, 0});
    return tensor.shuffle(dims);
}

Eigen::Tensor<float, 2> a_tensor(3, 4);
a_tensor.setRandom();

std::cout << "a_tensor is\n\n"<< a_tensor << "\n\n";
std::cout << "a_tensor transpose is\n\n"<< transpose(a_tensor) << "\n\n";

Later, when we discuss examples using tensors, we'll see some examples of geometric operations.softmax

8. Reduce

A reduction is a special case of operation that results in a tensor with a lower dimensionality than the original tensor. An intuitive case for reduction is:sum()maximum()

Eigen::Tensor<float, 3> X(5, 2, 3);
X.setRandom();

std::cout << "X is\n\n"<< X << "\n\n";

std::cout << "X.sum(): " << X.sum() << "\n\n";
std::cout << "X.maximum(): " << X.maximum() << "\n\n";

In the example above, we scaled down all sizes once. We can also perform reductions along specific axes. For example:

Eigen::array<int, 2> dims({1, 2});

std::cout << "X.sum(dims): " << X.sum(dims) << "\n\n";
std::cout << "X.maximum(dims): " << X.maximum(dims) << "\n\n";

The feature tensor API has a set of pre-built reduction operations such as , , , etc. If any of the pre-built operations are not suitable for a particular implementation, we can provide a custom functor as an argument.prodanyallmeanreduce(dims, reducer)reducer

Nine, tensor convolution

In a previous story , we learned how to implement 2D convolution using only vanilla C++ and the feature matrix. In fact, this is necessary because there is no built-in matrix convolution in Eigen. Fortunately, the EigenTensor API has a convenience function to perform convolution on EigenTensor objects:

Eigen::Tensor<float, 4> input(1, 6, 6, 3);
input.setRandom();

Eigen::Tensor<float, 2> kernel(3, 3);
kernel.setRandom();

Eigen::Tensor<float, 4> output(1, 4, 4, 3);

Eigen::array<int, 2> dims({1, 2});
output = input.convolve(kernel, dims);

std::cout << "input:\n\n" << input << "\n\n";
std::cout << "kernel:\n\n" << kernel << "\n\n";
std::cout << "output:\n\n" << output << "\n\n";

Note that we can perform 2D, 3D, 4D, etc. convolutions by controlling the dimensions of the slides in the convolution.

10. Soft maximum with tensor

When programming deep learning models, we use tensors instead of matrices. It turns out that matrices can represent one or at most two-dimensional grids, while we have higher-dimensional data multi-channel images or batch registers to process. This is where tensors come into play.

Let's consider the following example, where we have two batches of registers, each batch has 4 registers, and each register has 3 values:

We can represent this data as follows:

Eigen::Tensor<float, 3> input(2, 4, 3);
input.setValues({
    {
   
   {0.1, 1., -2.},{10., 2., 5.},{5., -5., 0.},{2., 3., 2.}},
    {
   
   {100., 1000., -500.},{3., 3., 3.},{-1, 1., -1.},{-11., -0.2, -.1}}
});

std::cout << "input:\n\n" << input << "\n\n";

Now, let's apply to this data:softmax

Eigen::Tensor<float, 3> output = softmax(input);
std::cout << "output:\n\n" << output << "\n\n";

Softmax is a popular activation function. We covered its implementation in a previous story . Now, let's introduce the implementation:Eigen::MatrixEigen::Tensor

#include <unsupported/Eigen/CXX11/Tensor>

auto softmax(const Eigen::Tensor<float, 3> &z)
{

    auto dimensions = z.dimensions();

    int batches = dimensions.at(0);
    int instances_per_batch = dimensions.at(1);
    int instance_length = dimensions.at(2);

    Eigen::array<int, 1> depth_dim({2});
    auto z_max = z.maximum(depth_dim);

    Eigen::array<int, 3> reshape_dim({batches, instances_per_batch, 1});
    auto max_reshaped = z_max.reshape(reshape_dim);

    Eigen::array<int, 3> bcast({1, 1, instance_length});
    auto max_values = max_reshaped.broadcast(bcast);

    auto diff = z - max_values;

    auto expo = diff.exp();
    auto expo_sums = expo.sum(depth_dim);
    auto sums_reshaped = expo_sums.reshape(reshape_dim);
    auto sums = sums_reshaped.broadcast(bcast);
    auto result = expo / sums;

    return result;
}

This code outputs:

We won't cover Softmax in detail here. If you need to check out the Softmax algorithm, don't hesitate to read the previous story again on Medium . For now, we'll just focus on understanding how to use feature tensors to encode our deep learning models.

The first thing to note is that this function does not actually calculate the softmax value of the parameters. In fact, only mount a complex object that can calculate the softmax.softmax(z)zsoftmax(z)

The actual value is only evaluated when the result of is assigned to a tensor-like object. For example, here:softmax(z)

Eigen::Tensor<float, 3> output = softmax(input);

Before this line, everything is just a calculation graph of softmax, hoping to be optimized. This happens only because we use keywords in the body of . Thus, the feature tensor API can optimize the entire computation using fewer operations, resulting in improved processing and memory usage.autosoftmax(z)softmax(z)

Before closing this story, I would like to point out and appeal to:tensor.reshape(dims)tensor.broadcast(bcast)

Eigen::array<int, 3> reshape_dim({batches, instances_per_batch, 1});
auto max_reshaped = z_max.reshape(reshape_dim);

Eigen::array<int, 3> bcast({1, 1, instance_length});
auto max_values = max_reshaped.broadcast(bcast);

reshape(dims)is a special geometric operation that produces another tensor with the same size as the original tensor, but with different dimensions. Reshape does not change the order of the data inside the tensor. For example:

Eigen::Tensor<float, 2> X(2, 3);
X.setValues({
   
   {1,2,3},{4,5,6}});

std::cout << "X is\n\n"<< X << "\n\n";

std::cout << "Size of X is "<< X.size() << "\n\n";

Eigen::array<int, 3> new_dims({3,1,2});
Eigen::Tensor<float, 3> Y = X.reshape(new_dims);

std::cout << "Y is\n\n"<< Y << "\n\n";

std::cout << "Size of Y is "<< Y.size() << "\n\n";

Note that, in this example, the size of X and Y is either 6 although they have very different geometry.

tensor.broadcast(bcast) repeats the tensor as many times as provided in the parameter for each dimension. For example:bcast

Eigen::Tensor<float, 2> Z(1,3);
Z.setValues({
   
   {1,2,3}});
Eigen::array<int, 2> bcast({4, 2});
Eigen::Tensor<float, 2> W = Z.broadcast(bcast);

std::cout << "Z is\n\n"<< Z << "\n\n";
std::cout << "W is\n\n"<< W << "\n\n";

Differently, the tensor rank (i.e., dimensionality) will not be changed, but only the size of the dimensionality will be increased.reshapebroadcast

11. Limitations

The Eigen Tensor API documentation cites some limitations that we can be aware of:

GPU support is tested and optimized for floating point types. Even if we could state that, the use of non-floating point tensors is discouraged when using the GPU.Eigen::Tensor<int,...> tensor;
The default layout (col-major) is the only one actually supported. At least for now we shouldn't be using row majors.
The maximum number of dimensions is 250. This size is only achievable when using a C++11 compliant compiler.

12. Conclusion and next steps

Tensors are the fundamental data structure of machine learning programming, allowing us to represent and process multidimensional data as directly as regular two-dimensional matrices.

In this story, we introduced the feature tensor API and learned how to use tensors with relative ease. We also learned that the Feature Tensor API has a lazy evaluation mechanism that optimizes execution in terms of memory and processing time.

To make sure we really understand the usage of the Eigen Tensor API, we walk through an example encoding Softmax using tensors.

In the next stories, we will continue to develop high-performance deep learning algorithms from scratch using C++ and Eigen, specifically using the Eigen Tensor API.

Thirteen, github code

You can find the code used in this story in this repository on GitHub .

14. Citation

[1] Feature Tensor API

[2] Feature tensor module

[3] Own Gitlab repository， libown / own · GitLab

[4] Charu C. Aggarwal， Neural Networks and Deep Learning： A Textbook （2018）， Springer

[5] Jason Brownlee，A Gentle Introduction to Tensors for Machine Learning with NumPy

About this series

In this series , we'll learn how to code must-know deep learning algorithms such as convolutions, backpropagation, activation functions, optimizers, deep neural networks, and more, using only plain and modern C++.