Crowd Density Estimation--Paper Reading: DM-Count

1. Paper address
2. GitHub open source code address

The following translations are mainly machine translations combined with manual proofreading.

Summary

In crowd counting, each training image contains multiple people, each marked with a dot. Existing crowd counting methods require smoothing each annotated point using Gaussian or estimating the likelihood of each pixel given an annotated point. In this paper, we show that imposing a Gaussian approach to annotations hurts generalization performance. Instead, we propose to use Distribution Matching for crowd counting (DM-Count). In DM-Count, we use Optimal Transport to measure the similarity between the normalized predicted density map and the normalized GT density map. To stabilize OT computation, we include a Total Variation loss in the model. We demonstrate that DM-Count has a tighter generalization error bound than that of Gaussian smoothing methods. In terms of mean absolute error (MAE), DM-Count substantially outperforms previous state-of-the-art methods on two large counting datasets (UCF-QNRF and NWPU), and achieves state-of-the-art results on ShanghaiTech and UCF-CC50 datasets. DM-Count reduces the error of the latest published results by about 16%.

1. Introduction

Image-based crowd counting is an important research problem with various applications in many domains including journalism and surveillance. Current state-of-the-art methods [54, 8, 25, 55, 61, 59, 17, 48, 21, 23, 36] treat crowd counting as a density map estimation problem, where a deep neural network first produces a given input image The 2D crowd density map of , and then estimate the total size of the crowd by summing the density values ​​at all spatial locations of the density map. For images of large crowds, this density map estimation method has been shown to be more robust than "detect before counting" methods [22, 19, 62, 12] because the former is less sensitive to occlusions and does not require early binning. Diversified decision-making.

A key step in the development of density map estimation methods is the training of deep neural networks that map from input images to corresponding annotated density maps. In all existing crowd counting datasets [15, 60, 14, 51], the annotated density map of each training image is a sparse binary mask in which each person’s head or forehead is Marked with a dot. Due to the laborious work required to describe the spatial extent, especially when the occlusion blur is too large, the spatial extent of each individual is not provided. Given training images with point annotations, training a density map estimation network is equivalent to optimizing the network parameters to minimize a differentiable loss function that predicts the loss density, which measures the difference between the predicted density map and the point annotation map. It is worth noting that the former is a dense real-valued matrix, while the latter is a sparse binary matrix. Given the sparsity of points, it is difficult to train a function defined based on the pixel-level differences between annotated and predicted density maps, since the reconstruction loss is very unbalanced between 0 and 1 in a sparse binary matrix. One way to alleviate this problem is to turn each annotated point into a Gaussian blob, so that the GT is more balanced and therefore the network is easier to train. Almost all previous crowd density map estimation methods [56, 57, 60, 38, 20, 33, 35, 4, 28, 50, 27, 40, 29, 26] follow this convention. Unfortunately, the performance of the resulting network is highly dependent on the quality of this "pseudo ground truth", but given the large variance in the size and shape of people in the perspective of crowded scenes, setting Getting the width right is not an easy task.

Recently, Ma et al. [31] proposed a Bayesian loss to measure the difference between predicted and annotated density maps. This method converts a binary GT annotation map into N "smooth GT" density maps, where N is the number of people. Given the location of that pixel, each pixel value of the smoothed GT density map is the posterior probability of the corresponding annotation point. Empirically, this method outperforms other aforementioned methods [60, 38, 20, 33, 35, 4]. However, there are two main problems with this loss function. First, it also requires a Gaussian kernel to construct the likelihood function for each annotated point, which involves setting the kernel width. Second, this loss corresponds to an uncertain system of equations with infinitely many solutions. The loss may be 0 for many density maps that are not similar to the GT density map. As a result, training with this loss may result in a predicted density map that is very different from the GT density map.

In this paper, we address the shortcomings of existing methods with the following contributions.
• We show theoretically and empirically that imposing Gaussian methods on annotations hurts the generalization performance of crowd counting networks.
• We recommend using DM-Count, a method for d-distribution matching for population counting. Unlike previous work, DM-Count does not require any Gaussian-smoothed GT annotations. Instead, we use optimal transfer (OT) to measure the similarity between the normalized predicted density map and the normalized GT density map. To stabilize OT computation, we further add a total variation (TV) loss.
• We give generalization error bounds for counting loss, OT loss, TV loss and overall loss in our method. All bounds are tighter than those of the Gaussian smoothing method.
• Empirically, our method greatly improves the state-of-the-art on four challenging crowd counting datasets: UCF-QNRF, NWPU, ShanghaiTech and UCF-CC50. Notably, our method reduces the published state-of-the-art MAE on the NWPU dataset by about 16%.

2. Previous work

2.1 Crowd counting method

Crowd counting methods can be divided into three categories: detect-then-count methods, direct regression counts, and density map estimation methods. Early methods [22, 19, 62, 12] can detect people, heads or upper bodies in images. However, accurate detection is difficult in dense crowds. Furthermore, it requires bounding box annotations, which is a laborious and ambiguous process due to severe occlusions. Later methods [5, 6, 49, 7] avoid the detection problem and directly learn to regress counts from feature vectors. But their results are difficult to interpret, and point-annotated plots are underutilized. Recent studies [20, 35, 15, 4, 31, 50, 27, 40, 29, 26, 54, 8, 25, 55, 61, 59, 17, 48, 21, 23, 30, 47, 53 43, 37, 56, 18, 39, 24, 44, 42] are estimated based on density maps, which have been shown to be more reliable than detection-then-count and count-regression methods.
Density map estimation methods typically define a training loss based on the pixel-wise difference between a Gaussian-smoothed and predicted density map. Instead of using a single kernel width to smooth point annotations, [60, 14, 47] use an adaptive kernel width. The kernel width is chosen based on the distance to the nearest neighbor of the annotation point. Specifically, [15] generated multiple smoothed GT density maps at different density levels. The final loss incorporates reconstruction errors from multiple density levels. However, these methods assume that the population is evenly distributed. In fact, the crowd distribution is very irregular. The Bayesian loss method [31] uses a Gaussian to construct a likelihood function for each annotated point. However, since the loss is not determined, it may not be possible to predict the correct density. A detailed analysis can be found in Section 4.2.

2.2 Optimal transmission

We propose a novel loss function based on Optimal Transport (OT) [46]. To better understand the proposed method, we briefly review the Monge-Kantarovich OT formulation in this section.

Optimal transportation refers to the optimal cost of transforming one probability distribution into another probability distribution. Let $\mathcal{X}={\left\{{\mathbf{x}}_i\mid {\mathbf{x}}_i\in {\mathbb{R}}^d\right\}}_{i=1}^n$和$\mathcal{Y}={\left\{{\mathbf{y}}_j|{\mathbf{y}}_j\in R^d\right\}}_{j=1}^n$for two $A\; set\; of\; points\; in\; a\; d-$ dimensional vector space. Let$\mu$和$\nu$ is two definitions in$\mathcal{X}$ 和 Y \mathcal{Y} A probability measure on$Y$$m,n\in {\mathbb{R}}_{+}^n$且$1_n^{T}m=1_n^{T}n=1(1_n$is a $n-$ dimensional all-ones vector). Let$c:\mathcal{X}\times \mathcal{Y}\mapsto R_{+}$is from point $X$ moves to point$The\; cost\; function\; of\; Y$ ,$C$ is the corresponding$n\times unique$ function:${\mathbf{C}}_{ij}=c\left({\mathbf{x}}_i,{\mathbf{y}}_j\right)$ . Let$\Gamma$ is to change the probability mass from$X$ passes to$The\; Y-$ shaped equation of the infinitive is:$C=\left\{c\in {\mathbb{R}}_{+}^{n\times n}:c1=m,c^{T}1=\nu \right\}$。 介于$\mu$和ν \nuThe Monge-Kantorovich optimal transmission (OT) cost between$\nu$
$$W(\mu ,n)=\underset{\gamma \in \Gamma }{\min }⟨\mathbf{C},\gamma ⟩$$
Intuitively, if the probability distribution $\mu$ is viewed as a unit amount of “dirt” piled on $\mathcal{X}$ and $\mu$ a unit amount of dirt piled on $Y$ , the OT cost is the minimum “cost” of turning one pile into the other. The OT cost is a principal measurement to quantify the dissimilarity between two probability distributions, also taking into account the distance between “dirt” locations. (This paragraph I don’t know how to translate it...)
The OT cost can also be calculated by the dual formula (2):
$$W(\mu ,n)=\underset{\alpha ,\beta \in R^n}{\max }⟨a,m⟩+⟨b,n⟩,\text{ st }{a}_i+b_j\le c\left({\mathbf{x}}_i,{\mathbf{y}}_j\right),\forall i,j$$

3. DM-Count: Distribution Matching for Crowd Counting

We consider crowd counting to be a distribution matching problem. In this section, we propose DM-Count: Distribution Matching for Crowd Counting, a network for crowd counting that inputs an image and outputs a map of density values. The final count estimate is obtained by summing the predicted density maps. DM-Count is independent of different network architectures. In our experiments, we use the same network as in the Bayesian loss paper [31]. Unlike all previous density map estimation methods that require a Gaussian to smooth GT case annotations, DM-Count does not require any Gaussian to preprocess GT annotations.

令$\mathbf{z}\in {\mathbb{R}}_{+}^n$represents a vectorized binary map of point annotations, and $\hat{z}\in R_{+}^n$Vectorized predicted density map returned by the neural network. By placing $\mathrm{z}$ 和 $\hat{\mathrm{z}}$ Treated as an unnormalized density function, we use the following three terms to represent the loss function in DM-Count: count loss, OT loss and total change (TV) loss. The first term measures the difference between the total masses, while the last two measure the difference between the normalized density function distributions.

Count loss. Let $\Vert \cdot {\Vert }_1$L 1 L_{1} representing the vector$L_1$norm, so $\Vert \mathbf{z}{\Vert }_1，\Vert \hat{z}{\Vert }_1$are ground truth and predicted counts, respectively. The goal of crowd counting is to make $\Vert \hat{z}{\Vert }_1$as close as possible to $\Vert z{\Vert }_1$, and then the counting loss is defined as the difference in absolute value between them:
$${\ell }_C(\mathbf{z},\hat{\mathbf{z}})=|\Vert \mathbf{z}{\Vert }_1-\Vert \hat{z}{\Vert }_1|$$
Optimal transmission loss. $z$和$\hat{z}$ are both unnormalized density functions, but we can convert them to probability density functions (pdfs) by dividing them by their respective sums. Besides OT, Kullback-Leibler divergence and Jensen-Shannon divergence can also measure the similarity between two pdfs. However, if the source and target distributions do not overlap, these measurements will not provide efficient gradients to train the network [32]. Therefore, we propose to use OT in this work. We define the OT loss as follows:
$${\ell }_{OT}(\mathbf{z},\hat{\mathbf{z}})=\mathcal{W}\left(\frac{\mathbf{z}}{\Vert \mathbf{z}{\Vert }_1},\frac{\hat{\mathbf{z}}}{\Vert \hat{z}{\Vert }_1}\right)=⟨a^{\ast },\frac{\mathbf{z}}{\Vert \mathbf{z}{\Vert }_1}⟩+⟨b^{\ast },\frac{\hat{z}}{\Vert \hat{z}{\Vert }_1}⟩$$
where$a^{\ast }$ and$b^{\ast }$ is formula$\left(2\right)$ solution. We use the quadratic transmission cost,$c(\mathbf{z}(i),\hat{z}(j))=\Vert \mathbf{z}(i)-\hat{z}(j){\Vert }_2^2,$ of which$z\left(i\right)$和$\hat{\mathbf{z}}\left(j\right)$iirespectively$i$ and$j$的$2Dcoordinates$ . To prevent division by zero errors, we add machine precision numbers to the denominator. Since$z$The elements in$\hat{}$$\hat{z}$ Definition:
$$\frac{\partial {\ell }_{OT}(\mathbf{z},\hat{z})}{\partial \hat{\mathbf{z}}}=\frac{b^{\ast }}{\Vert \hat{z}{\Vert }_1}-\frac{⟨b^{\ast },\hat{z}⟩}{\Vert \hat{z}{\Vert }_1^2}$$
This gradient can be backpropagated to learn the parameters of the density estimation network.

Total change loss. In each training iteration, we use the Sinkhorn algorithm [34] to approximate $a^{\ast }$ and$b^{\ast }$ . The time complexity is$O\left(n^2\log n/{ϵ}^2\right)$[9], where$ϵ$ is the desired optimal margin, the difference between the returned target and the optimal target. When optimizing with the Sinkhorn algorithm, the objective function drops sharply at the beginning but only slowly converges to the optimal solution in later iterations. In practice, we set a maximum number of iterations and the Sinkhorn algorithm returns only approximate solutions. As a result, when we optimize the OT loss using the Sinkhorn algorithm, the predicted density map ends up being close to the GT density map, but not quite the same. The OT loss will approximate dense areas of crowds well, but the approximation may be worse for low density areas of crowds. To address this, we additionally use the Total Variation (TV) loss, defined as:
$${\ell }_{TV}(\mathbf{z},\hat{\mathbf{z}})={\Vert \frac{\mathbf{z}}{\Vert \mathbf{z}{\Vert }_1}-\frac{\hat{z}}{\Vert \hat{z}{\Vert }_1}\Vert }_{TV}=\frac{1}{2}{\Vert \frac{\mathbf{z}}{\Vert \mathbf{z}{\Vert }_1}-\frac{\hat{z}}{\Vert \hat{\mathbf{z}}{\Vert }_1}\Vert }_1$$
The TV loss will also increase the stability of the training process. Optimizing OT loss with the Sinkhorn algorithm is a minimum-maximum saddle point optimization process, similar to GAN optimization [13]. As shown in Pix2Pix GAN [16], the stability of GAN training can be increased by increasing the reconstruction loss. To this end, the TV loss is similar to the reconstruction loss and also adds stability to the training process. TV loss vs predicted density map $\hat{z}$ Definition:
$$\frac{\partial {\ell }_{TV}(\mathbf{z},\hat{\mathbf{z}})}{\partial \hat{z}}=-\frac{1}{2}\left(\frac{\mathrm{sign}(\mathbf{v})}{\Vert \hat{z}{\Vert }_1}-\frac{⟨\mathrm{sign}(\mathbf{v}),\hat{z}⟩}{\Vert \hat{z}{\Vert }_1^2}\right)$$
其中，$\mathbf{v}=\mathbf{z}/\Vert \mathbf{z}{\Vert }_1-\hat{z}/\Vert \hat{z}{\Vert }_1,$ ，$sign(\cdot )$ is the sign function on each element of the vector.
The overall loss function (The Overall Objective). The overall loss function is the sum of count loss, OT loss and TV loss:
$$\ell (\mathbf{z},\hat{\mathbf{z}})={\ell }_C(\mathbf{z},\hat{z})+l_1{\ell }_{OT}(z,\hat{z})+l_2\Vert \mathbf{z}{\Vert }_1{\ell }_{TV}(z,\hat{z})$$

where $l_1$and $l_2$are tunable hyperparameters for OT and TV losses. To ensure that the TV loss has the same proportion as the count loss, we multiply this loss term with the total headcount.

Given $K$ training images{ I k } k = 1 K \left \{I_ {k} \right \} _ {k = 1} ^ {K}{ Ik​}k=1K​and its corresponding point annotation graph { zk } k = 1 K \left \{\mathbf {z} _ {k} \right \} _ {k = 1} ^ {K}{ zk​}k=1K​, we will minimize $L(f)=\frac{1}{K}{\sum }_{k=1}^{\bar{K}}\ell \left({\mathbf{z}}_k,f\left(I_k\right)\right)$ to learn a deep neural network fffor density map estimation$f$ 。

4. Generalization Boundary and Theoretical Analysis

In this section, we analyze the Gaussian smoothing method, the Bayesian loss and the theoretical properties of the proposed DM-Count. Proofs of the theorems in this section can be found in the supplementary material. First, we introduce some notation below.

令D = { ( I , z ) } \mathcal {D} = \{(I,\mathbf {z})\}D={ (I,z ) } is the joint distribution of crowd images and corresponding point annotation maps. Let$H$ is the hypothesis space. Every$h\in H$ -hold$I\in Iprojection$ z ∈ Z \mathbf {z }$\mathbf{z}\in$Each dimension of$Z.$Let$\mathcal{F}=\mathcal{H}\times \cdots \times H\left(n\text{ times}\right)x$ is the mapping space. Every$f\in \mathcal{F}$ 把 $I\in Iprojection$ z ∈ Z \mathbf {z }$\mathbf{z}\in Z.$ _ Suppose$t$ for each$z\in$Gaussian smoothed density map in$D$ , and let D ~ = { ( I , t ) } \tilde{\mathcal {D}} = \{(I,\mathbf {t})\}D~={ (I,t)} 为 $(I,t)$ joint distribution. LetS = { ( I k , zk ) } k = 1 KS = \left \{\left(I_ {k},\mathbf {z} _ {k} \right)\right \} _ {k = 1} ^{K}S={ (Ik​,zk​)}k=1K​ 和 $\tilde{S}={\left\{\left(I_k,{\mathbf{t}}_k\right)\right\}}_{k=1}^K$for containing $A\; finite\; set\; of\; K$ samples, and respectively from$D$和$\mathcal{D}$The independent and identically distributed samples sampled in$\stackrel{~\; .}{}$Suppose$R_S\left(H\right)$ means$H$ aboutSSThe empirical Rademacher complexity of$S\; (empirical\; Rademacher\; complexity)\; [3].$Given a data setD ∈ { D , S , D ~ , S ~ } D \in \{\mathcal {D} ,S,\tilde {\mathcal {D}},\tilde {S} \}D∈{ D,S,D~,S~ }, the mapping$f\in F$ and loss function$\ell$ , 令$\mathcal{R}(D,f,\ell )={\mathbb{E}}_{(I，\mathbf{s})\sim D}[\ell (\mathbf{s},f\left(I\right)\left)\right]$ defines the graphical function and${\ell }_1(\mathbf{z},\hat{\mathbf{z}})=\Vert \mathbf{z}-\hat{z}{\Vert }_1$ 。设 $f_D^D={\mathrm{argmin}}_{f\in \mathcal{F}}\mathcal{R}\left(D,f,{\ell }_{}\right)$ is the loss${\ell }_{}$In dataset DDThe minimum value R on$D$$\mathcal{R}\left(D,f,{\ell }_{}\right)$ , whereD ∈ { D , S , D ~ , S ~ } D \in \{\mathcal {D},S,\tilde {\mathcal {D}},\tilde {S} \}D∈{ D,S,D~,S~ }，Δ ∈ { 1 , C , OT , TV , ∅ } \Delta \in \{1,C,OT,TV,\emptyset \}D∈{ 1,C,OT,TV,∅} 。

4.1 Generalized Error Bounds for Gaussian Smoothing Method

Many existing methods (e.g. [60, 20, 35]) use Gaussian smoothed annotation maps for training. Using ℓ 1 \ell_1 on the density plot is given below${\ell }_1$The generalization error bound for the loss.
Theorem 1 : Suppose $\forall f\in F$且$(I,\mathbf{t})\sim \tilde{\mathcal{D}}$,有 $\ell (\mathbf{t},f(I))\le B.$ Then, for any$0<d<1$ , at least with probability$1-\delta$有,
a) 泛化误差的上边界为：
$$\mathcal{R}\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)\le R\left(\tilde{\mathcal{D}},f_1^{\tilde{D}},{\ell }_1\right)+2n{R}_S(\mathcal{H})+5B\sqrt{2\log (8/d)/K\_}+{\mathbb{E}}_{(I,\mathbf{z})\sim \mathcal{D}}\Vert \mathbf{z}-\mathbf{t}{\Vert }_1$$
b) 泛化误差的下边界为：
$$\mathcal{R}\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)\ge \left|{\mathbb{E}}_{(I,z)\sim D}\Vert z-t{\Vert }_1-R\left(\tilde{\mathcal{D}},f_1^{\tilde{S}},{\ell }_1\right)\right|$$
In this theorem, as the number of samples $K$ region is infinite,$2n{R}_S\left(H\right)$ and$5B\sqrt{2\log (8/d)/K\_}$drop to $0$ . Theorem$1.a$ ) shows that the expected risk$\mathcal{R}\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)$ , which is evaluated on real GT data using empirical minima trained on Gaussian-smoothed GT. Under the condition of sufficient training data, no more than$\mathcal{R}\left(\tilde{\mathcal{D}},f_1^{\tilde{D}},{\ell }_1\right)+{\mathbb{E}}_{(I,\mathbf{z})\sim \mathcal{D}}\Vert \mathbf{z}-\mathbf{t}{\Vert }_1$Theorem $1.b$ ) Show that the expected risk$R\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)$ is not less than$\left|{\mathbb{E}}_{(I,\mathbf{z})\sim \mathcal{D}}\Vert \mathbf{z}-\mathbf{t}{\Vert }_1-\mathcal{R}\left(\tilde{\mathcal{D}},f_1^{\tilde{S}},{\ell }_1\right)\right|.Let$ function$R\left(\tilde{D},f_1^{\tilde{S}},{\ell }_1\right)\le E_{(I,z)\sim D}\Vert z-t{\Vert }_1,$ so the smallest is$\mathcal{R}\left(\tilde{\mathcal{D}},f_1^{\tilde{S}},{\ell }_1\right)$ , the maximum expected risk is$R\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)$ . In other words, the model$f_1^{\tilde{S}}$In Gaussian smoothed GT $\stackrel{The\; better\; the\; performance\; on\; ~}{D}$ , in the actual GTD \mathcal{D}The generalization performance on$D$$R\left(\tilde{D},f_1^{\tilde{S}},{\ell }_1\right)\ne {\mathbb{E}}_{(I,\mathbf{z})\sim \mathcal{D}}\Vert \mathbf{z}-\mathbf{t}{\Vert }_1,$ 当$\mathcal{R}\left(\tilde{\mathcal{D}},f_1^{\tilde{S}},{\ell }_1\right)=0.$，有 $R\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)>0.\mathcal{R}\left(D,f_1^{\tilde{S}},{\ell }_1\right)$ deterministic function${\mathbb{E}}_{(I,z)\sim D}\Vert z-t{\Vert }_1$. This is not what we want, because we want to evaluate the risk $\mathcal{R}\left(\mathcal{D},f_1^{\tilde{S}},{\ell }_1\right)$ tends to$0$。

4.2 Uncertain Bayesian loss

The Bayesian loss [31] is:

$${\ell }_{\text{Bayesian}}(\mathbf{z},\hat{\mathbf{z}})=\sum _{i=1}^N\left|1-⟨{\mathbf{p}}_i,\hat{z}⟩\right|,\text{ where }{\mathbf{p}}_i=\frac{N\left(q_i,p^2{1}_{2\times 2}\right)}{{\sum }_{i=1}^N\mathcal{N}\left({\mathbf{q}}_i,p^2{1}_{2\times 2}\right)}$$
where $N$ is$Foreach,\; N\; (\; qi\; ,\; \sigma \; 2\; 1\; 2\; \times 2\; )\; \backslash mathcal\{N\}\backslash$ $N\left(q_i,p^2{1}_{2\times 2}\right)$ is based on${\mathbf{q}}_i$is the mean, and the variance is $p^2{1}_{2\times 2}$Gaussian distribution. $q_i$is z \mathbf{z}the ithi^{th}thin$z$$i^{th}$ label points. .pi${\mathbf{p}}_i$和$The\; dimensions\; of\; z$ are$nis$ the number of pixels in the density map. However, due to the number of marked points$N$ is less than$n,$ the Bayesian loss is undefined. For GT annotation$z$ , there are infinitely many$\hat{\mathbf{z}}$ interface${\ell }_{\text{Bayesian}}(\mathbf{z},\hat{z})=0$且$\hat{z}\ne z.$ Therefore, the predicted density map may be very different from the GT density map.

4.3 Generalization Error Bounds for Loss Functions in DM-Count

In the following theorem, we give generalization error bounds for the loss function of the proposed method.
Theorem 2 : Suppose $\forall f\in F$且$(I,\mathbf{z})\sim D$ ,if$\Vert \mathbf{z}{\Vert }_1\ge 1,\Vert f(I){\Vert }_1\ge 1$ (can be obtained by adding$\mathrm{z}$ 和 $f\left(I\right)$ all add a virtual dimension with a value of 1 to satisfy) and${\ell }_C(\mathrm{z},f(I))\le B.$ Then, for any$0<d<1$ , at least with probability$1-With\; \delta$ :
$a.)$ The generalization error bound for the counting loss is:
$$\mathcal{R}\left(\mathcal{D},f_C^S,{\ell }_C\right)\le \mathcal{R}\left(D,f_C^{\mathcal{D}},{\ell }_C\right)+2n{R}_S(\mathcal{H})+5B\sqrt{2\log (8/d)/K\_}$$
$b\_)$ OT is independently defined as:
$$R\left(D,f_{OT}^S,{\ell }_{OT}\right)\le R\left(D,f_{OT}^D,{\ell }_{OT}\right)+4{\mathbf{C}}_{\infty }{n}^2{R}_S(\mathcal{H})+5{CORE}_{\infty }\sqrt{2\log (8/d)/K\_}$$
$c.)$ The generalization error bound for TV loss is:
$$\mathcal{R}\left(\mathcal{D},f_{TV}^S,{\ell }_{TV}\right)\le R\left(D,f_{TV}^{\mathcal{D}},{\ell }_{TV}\right)+n^2{R}_S\left(H\right)+5\sqrt{2\log (8/d)K\_}$$
$d\_)$ Specifies the equivalence of the following function:
$$\begin{array}{r}\mathcal{R}\left(\mathcal{D},f^S,\ell \right)\le R\left(D,f^{\mathcal{D}},\ell \right)+\left(2n+4{min}_1{\mathbf{C}}_{\infty }{n}^2+l_{2}N{n}^2\right)R_S(\mathcal{H})\\ +5\left(B+l_1{\mathbf{C}}_{\infty }+l_{2}N\right)\sqrt{2\log (8/d)K\_}\end{array}$$
where, $C_{\infty }$is the maximum value of the OT cost matrix, N = sup ⁡ { ∥ z ∥ 1 ∣ ∀ ( I , z ) ∼ D } N=\sup \left\{\|\mathbf{z}\|_{1} \mid \forall(I, \mathbf{z}) \sim \mathcal{D}\right\}N=sup{ ∥z∥1​∣∀(I,z)∼D } is the largest number of heads on the data set.
In the above theory, with$K$ increases,$R_S\left(H\right)$ and$\sqrt{2\log (1/d)K\_}$decrease. All using empirical minimizers $f_D^{\dot{S}}$The expected risk $\mathcal{R}\left(\mathcal{D},f_D^S,{\ell }_{}\right)$ converges to using optimal minimizers$f_D^{\mathcal{D}}$The expected risk $R\left(D,f_D^D,{\ell }_{}\right)$ ， Δ ∈ { C , O T , T V , ∅ } \Delta \in\{C, O T, T V, \emptyset\} D∈{ C,OT,TV,∅ } . This means that all caps are tight. Furthermore, all upper bounds are better than Theorem$1.$ The Gaussian smoothing method shown in$a$$)\; has\; a\; tighter\; upper\; bound.$Theorem$2.b$ ) The bound of OT loss and the maximum transmission cost${\mathrm{C}}_{\infty }$related. Therefore, we need to use a small transfer cost in OT for better generalization performance. $R_S\left(H\right)$ counting loss factor is$O\left(n\right)$ , the coefficient of OT loss and TV loss is$O\left(n^2\right)$. This means that for larger image sizes, we need more images for training. When using counting loss, this number is related to$The\; size\; of\; z$ has a linear relationship, while when using OT loss or TV loss, it is a quadratic relationship. When using these three loss functions, we need to$l_1$and $l_2$Set to be small in order to balance the three loss values.

5. Experiment

In this section, we present experiments on toy data and benchmark crowd counting datasets. A more detailed dataset description, implementation details and experimental setup can be found in the supplementary material.

5.1 Results on the Toy Data dataset

To understand the empirical behavior of different methods, we consider a toy problem in which the task is to convert the source density map z ^ \mathbf{\hat{z}} using pixel-wise loss, Bayesian loss and DM-Count$\hat{\mathbf{z}}$ Move to target density map$z$ . Source density map$\hat{z}$ is from$0$ to0.01 0.01Initialized with a uniform distribution between$0.0$$and$$1\; ,\; the\; target\; density\; map\; is\; shown\; on\; the\; far\; left\; in\; Figure\; 1$$.$All three methods start from the same source density map. Figure 1 shows z ^ \mathbf{\hat{z}}at the final convergence$\hat{\mathbf{z}}$ Visualization diagram. A pixel-wise loss produces a blurred density map with higher counts. The Bayesian loss performs better than the pixel-level loss [52] in terms of counting error, peak signal-to-noise ratio (PSNR), and structural similarity in images (SSIM) [52], but the final density map differs from the target value is very different, with high values ​​in many locations where there are no annotation points. This confirms our analysis that the Bayesian loss corresponds to an uncertain system and thus the output density map may differ significantly from the target density map. In contrast, DM-Count was able to produce more accurate counts and density maps. DM-Count significantly outperforms Bayesian loss in both PSNR and SSIM.

5.2 Results on the Benchmark dataset

We conduct experiments on four challenging crowd counting datasets: UCF-QNRF [15], NWPU [51], ShanghaiTech [60] and UCF-CC-50 [14]. Notably, the NWPU dataset is the largest and most challenging crowd counting dataset publicly available today. GT counts for test images are not released, and results on the test set must be obtained by submitting to the evaluation server https://www.crowdbenchmark.com/nwpucrowd.html . Prior to previous work [35, 15, 4, 14, 60], we used the following metrics: mean absolute error (MAE), root mean square error (RMSE) and mean normalized absolute error (NAE) as evaluation metrics. For all three metrics, smaller is better. For a fair comparison, we use the same network as in the Bayesian loss paper [31]. In all experiments, we set $l_1=0.1，l_2=0.01$ , and set the Sinkhorn entropy regularization parameter to 10. The number of Sinkhorn iterations was set to 100. On average, the OT computation time per image is 25ms.

Quantitative results. Tables 1 and 2 compare the performance of DM-Count with various methods. In all experiments, except the CAN method, DM-Count outperforms all other methods in MSE performance (based on the comparable NWPU dataset). Although we use the same hyperparameter set for DM-Count in all experiments, DM-Count still achieves the best performance, which shows that the performance of DM-Count is stable in various datasets.

In all experiments, DM-Count outperforms both pixel-wise and Bayesian losses when under the same network architecture and training procedure as DM-Count. This demonstrates the effectiveness of our proposed loss function. Pixel-wise loss is much worse than DM-Count in Table 1. Furthermore, DM-Count still achieves state-of-the-art performance on all four datasets even without using multi-scale architectures in [4, 47] or deeper networks in [2, 50]. This shows that it is very important to have a good loss function in crowd counting.

DM-Count significantly outperforms state-of-the-art methods on the large and challenging datasets of UCF-QNRF and NWPU. Specifically, on the UCF-QNRF dataset, DM-Count reduces the MAE and MSE of the Bayesian loss from 88.7 to 85.6 and from 154.8 to 148.3, respectively. It is worth noting that on the NWPU test set (obtained by submitting to the evaluation server), DM-Count greatly reduces MAE and NAE, MAE is reduced from 105.4 to 88.4, and NAE is reduced from 0.203 to 0.169.

qualitative results. Figure 2 shows the predicted density plots of pixel-wise loss, Bayesian loss and DM-Count. The figure shows that: 1) the count value produced by DM-Count is closer to the actual value of GT, and 2) the density map produced by DM-Count is clearer than pixel-wise loss and Bayesian loss. In Fig. 2, DM-Count produces much higher PSNR and SSIM than pixel-wise and Bayesian losses. On the entire UCF-QNRF test set, the average PSNR and SSIM for pixel-wise loss are 34.79 and 0.43, Bayesian loss is 34.55 and 0.42, and DM count is 40.65 and 0.55. Since the pixel-wise loss uses a Gaussian-smoothed ground truth, it produces densities that are blurrier than the actual ground truth. This empirically validates our theoretical analysis of the generalized bound on the Gaussian smoothing method. As shown, Pixelwise and Bayesian loss fail to locate people in densely populated areas. In contrast, DM-Count does a good job of locating people in both densely populated and sparsely populated areas. Figure 3 shows the predicted density map by DM counts. The predicted density maps are in good agreement with the population densities in both sparse and dense areas, demonstrating the effectiveness of DM-Count in spatial density estimation.

5.3 Model Simplification Test

Hyperparameter research. We adjust λ 1 λ1 in DM-Count of UCF-QNRF dataset$\lambda 1$和$\lambda 2$ . First, we will$l_1$fixed at 0.1, and $l_2$Adjusted from 0.01, 0.05 to 0.1. MAE varies from 85.6, 87.8 to 88.5. Since $l_2=0.01$ can get the best result, we will$l_2$fixed at 0.01, and $l_1$Adjusted from 0.01, 0.05 to 0.1. MAE varies from 87.2, 86.2 to 85.6. Therefore, we set $l_1=0.1，l_2=0.01anduse$ it for all datasets.

The effect of the number of Sinkhorn iterations. Table 3 lists the results of DM-Count on the UCF, QNRF datasets using different numbers of Sinkhorn iterations. As shown in the table below, using a small number of iterations degrades the performance of DM-Count, which indicates that the OT solution we obtain is not accurate. When the number of iterations increases to 100, the performance of DM-Count exceeds the previous state-of-the-art. The performance stabilizes after the number of iterations exceeds 100. Therefore, in all our experiments we use 100 Sinkhorn iterations for DM-Count.

Contribution of each loss function. DM-Count counting loss function consists of three parts, counting loss, OT loss and TV loss. We investigate the contribution of each function on the UCF-QNRF dataset. The results are listed in Table 5. As shown in the table, all components are essential for the final performance. However, OT loss is the most important component.

Robustness to noisy annotations. Crowd annotation is done by placing a point on the person. Such a process is ambiguous and can lead to unavoidable annotation errors. We study the performance of different loss functions for annotation errors. We add uniform random noise to the original annotations and train different models with the same noisy annotations. Noise is randomly generated between 0% and 5% of the image height, with an average of about 80 pixels. As shown in Table 4, the proposed DM-Count is more robust to annotation errors compared to the pixel-wise Bayesian loss.

6. Conclusion

In this paper, we have shown that smoothing GT point annotations with a Gaussian kernel hurts the generalization range of the model when tested on real GT data. Instead, we treat crowd counting as a distribution matching problem and propose optimized transport based DM-Count to address this problem. Unlike previous work, DM-Count does not require a Gaussian kernel to smooth annotated points. The generalization error bound of DM-Count is tighter than that of the Gaussian smoothing method. Extensive experiments on four crowd counting benchmarks demonstrate that DM-Count significantly outperforms previous state-of-the-art methods.