A Generalized Loss Function for Crowd Counting and Localization reading notes

To put it simply, UOT is used to solve the crowd counting problem.

Code: https://github.com/jia-wan/GeneralizedLoss-Counting-Pytorch.git
I changed it a little: https://github.com/Nightmare4214/GeneralizedLoss-Counting-Pytorch.git

loss

设density map为 A = { ( a i , x i ) } i = 1 n \mathcal{A} =\left\{\left(a_i, \mathbf{x}_i\right)\right\}_{i=1}^{n} A={ (ai​,xi​)}i=1n​
inside $a_i$To predict density, $x_i\in {\mathbb{R}}^n$ is the coordinate,$n$ is the number of pixels
, let$a={\left[a_i\right]}_i$, that is, the density map is converted into a column vector

The real point diagram is B = { ( bj , yj ) } j = 1 m \mathcal{B}=\left\{\left(b_j,\mathbb{y}_j\right)\right\}_{j=1 }^mB={ (bj​,yj​)}j=1m​
where $y_j$is the coordinate, $m$ is the number of labeled points,$b_j$For the number of people represented by this point,
this paper assumes that $b={\left[b_j\right]}_j=1_m$, that is to say, there is only one person at each point

Properties of the UOT
$${\mathcal{L}}_{\mathbf{C}}^{}\left(\mathcal{A},\mathcal{B}\right)=\underset{\mathbf{P}\in {\mathbb{R}}_{+}^{n\times m}}{\min }⟨\mathbf{C},\mathbf{P}⟩-ϵH\left(\mathbf{P}\right)+\tau D_1\left(P{1}_m|\mathbf{a}\right)+\tau D_2\left({\mathbf{P}}^T{1}_n|\mathbf{b}\right)$$
其中$\mathbf{C}\in {\mathbb{R}}_{+}^{n\times m}$is the transmission cost matrix, $C_{i,j}$To change the density from ${\mathbf{x}}_i$Move to ${\mathbf{y}}_j$The distance
$P$的电影电影最好
令$\hat{\mathbf{a}}=P1{\_}_m,\hat{\mathbf{b}}={\mathbf{P}}^T{1}_n$

This loss has 4 parts.
The first part is the transmission loss, which aims to move the predicted density map closer to the real label. The
second part is the entropy $H\left(\mathbf{P}\right)=-{\sum }_{i,j}{P}_{i,j}\log P_{i,j}$Is the entropy regularization term, which is used to control the degree of sparsity. The larger it is, the sparser it is (it will tend to be evenly distributed), and vice versa.

The third part is hope $\hat{\mathbf{a}}$ near$The\; fourth\; part\; of\; a$
is hope$\hat{\mathbf{b}}$ near$b$

In the paper, $D_1$Take $L_2$The square of
$D_2$Take $L_1$

cost matrix

$$C_{i,j}=e^{\frac{1}{h\left(x_i,y_j\right)}\Vert {\mathbf{x}}_i-{\mathbf{y}}_j{\Vert }_2}$$
Given the values ​​xi ${\mathbf{x}}_i,{\mathbf{y}}_j$It is normalized
, but please note that $the\left(x_i,y_j\right)$ is a constant, the default is$0.6$

Solve

Using the sinkhorn
$$\mathbf{P}=\mathrm{diag}(\mathbf{u})\mathbf{K}\mathrm{diag}\left(v\right),\mathbf{K}=\exp (-C/\epsilon )$$
here approximates$D_1,D_2$For the KL function, the equivalent of the smooth function
$${\mathbf{u}}^{(\ell +1)}={\left(\frac{\mathbf{a}}{\mathbf{K}{\mathbf{v}}^{(\ell )}}\right)}^{\frac{}{\tau +ϵ}},{\mathbf{v}}^{(\ell +1)}={\left(\frac{\mathbf{b}}{{\mathbf{K}}^{\top }{\mathbf{u}}^{(\ell +1)}}\right)}^{\frac{}{\tau +ϵ}}$$

(In fact, even if it is $KL$ divergence, other codes cannot seem to be written like this)

code

data set

preprocessing

Used UCF-QNRF

Preprocessing:
1. Let $h,$The smaller one in$w$$\left[512,2048\right]$ range, and the other is adjusted according to the scaling ratio
2. Filter points that are not in the picture
3. Additional calculation of a distance from each point to other points, specifically
$$\mathbf{P}=\left(\begin{array}{c}{\mathbf{p}}_1^T\\ p_2^T\\ ⋮\\ p_m^T\end{array}\right),p_i\in {\mathbb{R}}^2$$
$$\mathbf{dis}={\left[\Vert {\mathbf{p}}_i-{\mathbf{p}}_j\Vert \right]}_{i,j}$$

Finally, perform the process of selecting sentinels in quick sorting for each row, and find the 3rd (counting from 0)
pair of $1,2,$Average 3 elements (counting from 0$)$

def find_dis(point):
    a = point[:, None, :]
    b = point[None, ...]
    dis = np.linalg.norm(a - b, ord=2, axis=-1)  # dis_{i,j} = ||p_i - p_j||
    # mean(4th_min, 2 of the [1st_min, 2nd_min, 3rd_min])
    dis = np.mean(np.partition(dis, 3, axis=1)[:, 1:4], axis=1, keepdims=True)
    
    return dis

因此得到的标签为
$$\mathbf{P}={\left[\left(x_i,y_i,di{s}_i\right)\right]}_i\in {\mathbb{R}}^{m\times 3}$$

Read data

Randomly crop the picture to $\left(512,512\right)$
设$i,j$ is the coordinate of the upper left corner of cropping,$h=w=512$

Then read the tag
according to $dis$ to set a small rectangle.
Calculate the area of ​​this rectangle in the clipping range, and$\frac{1}{4}$
If this ratio is greater than 0.3, select this point, otherwise discard it

, and then the others are randomly flipped horizontally.

Model

vgg19+upsampling+two layers of convolution+abs

train

Note that sinkhorn here has $ϵ-\text{Scaling heuristic}$ can achieve convergence within 20 rounds.log-domain \text{log-domain}
is also used.$\text{log-domain}$

result

The results of the model provided by the author: mae 85.09911092883813, mse 150.88815648865386
The results I ran on UCF-QNRF: mae:85.69232401590861, mse:155.30853159819492