The amount of parameters (Params) and calculation (FLOPs) of the neural network

definition

Amount of parameters (Params)

The number of parameters refers to the total number of parameters that need to be trained during model training. Used to measure the size of the model (computational space complexity).

Calculations (FLOPs)

The number of floating-point operations, understood as the amount of calculation (calculation time complexity), can be used to measure the complexity of the algorithm, and is often used as an indirect measure of the speed of the neural network model (although recent articles have proved that the speed of the model is evaluated by the indirect standard of FLOPs It is unreliable, because the calculation speed of the model is also related to factors such as memory throughput, but this standard is still widely used as a reference evaluation standard for model speed). When counting FLOPS, we usually count addition, subtraction, multiplication, division, curtaining, square root, etc. as a single FLOP.

Formula representation

convolutional layer

输入维度：$W_{in}\ast H_{in}\ast C_{in}$
输出维度：$W_{out}\ast H_{out}\ast C_{out}$
Convolution kernel: $k_w\ast k_h$

参数量： $k_w\ast k_h\ast C_{in}\ast C_{out}$
Parameter quantity (including bias): $(k_w\ast k_h\ast C_{in}+1)\ast C_{out}$

计算量： $k_w\ast k_h\ast C_{in}\ast W_{out}\ast H_{out}\ast C_{out}$
Computation amount (considering addition and offset): $[(k_w\ast k_h\ast C_{in})+({\mathbf{k}}_w\ast k_h\ast C_{in}-1)+1]\ast W_{out}\ast H_{out}\ast C_{out}$

PS: Among them $k_w\ast k_h\ast C_{in}$Indicates the amount of multiplication, $k_w\ast k_h\ast C_{in}-1$ means addition calculation amount, +1 means bias

Pooling and activation layers

Activation and pooling layers just do a transformation of the original matrix and don't introduce new parameters. These other layer types definitely take time, but they don't use dot products and become a rounding error in the total computational complexity of the network.

fully connected layer

The fully connected layer sets the number of neurons in the next layer, and uses affine transformation $y_i={\vec{W}}_i\cdot \vec{x}+b_i$Get the value of the next layer of neurons, because the neurons between the two layers will all be connected, which is called full connection, as shown in the figure.

Input dimension: $d_{in}$
Output dimension: $d_{out}$

参数量： $d_{in}\ast d_{out}$
Parameter amount (including bias): $(d_{in}+1)\ast d_{out}$

计算量： $d_{in}\ast d_{out}$
Computation amount (consider addition and offset): $[d_{in}+(d_{in}-1)+1]\ast d_{out}$
PS： $d_{in}-1$ represents the amount of addition operation (the addition operation required to multiply the weight matrix and the matrix vector of the input/previous layer value), +1 represents the bias, and the input is a multidimensional direct multiplication.

BN layer

The parameters introduced by the BN layer are related to the number of neurons in the input layer. Assuming that the number of input neurons is n , the parameters introduced by this layer are 2n . However, in some cases, the influence of the Batch Normalization layer can be completely ignored.

Summary: For the neural network model, the reduction of network parameters should be mainly aimed at the fully connected layer; when optimizing the amount of calculation, the focus should be on the convolutional layer.

Supplement: In the written test, sometimes the size of the output feature map is not given, and you need to calculate it yourself. The calculation of the size of the feature map is as follows:

卷积层：$ou{t}_{size}=\frac{InPutSize-KernelSize+2\ast Padding}{Stride}+1$

池化层：$ou{t}_{size}=\frac{InPutSize-KernelSize}{Stride}+1$

PS: When the calculation result is not an integer, the convolution layer is rounded up, and the pooling layer is rounded down.