UFLD & UFLDv2 paper study

UFLD & UFLDv2 Learning

FLD

The judgment of the lane position is performed in units of behavior, that is, the lane is detected based on the anchor.

A New Paradigm for Lane Marking Detection

The author believes that the problems of high speed and no visual information are extremely important in lane line detection, and the proposed method effectively solves these problems.

Paradigm Definition : To address the above issues, the authors propose to formulate lane detection as a line selection method based on global image features. In other words, use global features to select the correct lane location on each predefined row. In the formula, a lane is represented as a series of horizontal positions of a predefined row, known as row anchors. To represent locations, the first step is meshing. On each row anchor, the position is divided into a number of cells. In this way, channel detection can be described as selecting certain cells on predefined row anchors, as shown in Fig. 3(a).

Let the maximum number of lanes be C, the number of row anchor points be h, and the number of grid cells be w. Let X be the global image feature, $f^{ij}$ is the$i$ lane line,$A\; classifier\; for\; selecting\; lane\; locations\; on\; j$ row anchors. Then lane prediction can be written as:
$$P_{i,j,:}=f^{ij}(X),s.t.i\in [1,C],j\in [1,h],$$
of which$P_{i,j,:}$Because $(w+1)$ dimensional vector representing the choice$(w+1)$ grid cell as the$i$ lane, the$Probability\; of\; row\; j$ anchors. Suppose$T_{I,J,:}$is a one-hot coded label in the correct position. The loss function of this paradigm is:
$$L_{cls}=\sum _{i-1}^C\sum _{j=1}^h{L}_{CE}(P_{i,j,:},T_{i,j,:}),$$
where$L_{CE}$is the cross-entropy loss. Use an extra dimension to represent the absence of lanes, so $(w+1)$ dimension instead of$w$ dimension for classification.

PS: The author believes that this method predicts the probability distribution of all positions on each row anchor based on the global features. Therefore, the correct location can be selected according to the probability distribution.

How to achieve faster running speed : The difference between the proposed paradigm and segmentation is shown in Fig. 3. Suppose the image size is $H\times w$ . In general, the number of predefined row anchors and the size of the grid are much smaller than the size of an image, that is to say$h\ll H$和$w\ll w$ . In this way, the original division formula needs to be carried out$(C+1$ ) Dimension$H\times W$ is classified, and our formula only needs to solve$(W+1)$ Dimension$C\times H$ classification problem. This can greatly reduce the computational size, because the computational cost of our formula is$C\times h\times (w+1)$ , and the computational cost of the split is$H\times w\times (C+1)$ . For example, using the general setting of the CULane dataset [22], the ideal computational cost of our method is$1.7\times 10^4$ calculations, the calculation cost of division is$1.15\times 10^6$ calculations. Computational costs are greatly reduced, and our formula can achieve extreme speed.

How to deal with the no-visual-cues problem : To deal with the no-visual-cues problem, it is important to utilize information from other locations, since no visual cues means no information at the target location. For example, a lane is blocked by a car, but we can still locate the lane with information about other lanes, road shape, and even the direction of the car. Therefore, exploiting information from other locations is the key to solving the problem of no visual cues, as shown in Figure 1.

From the perspective of the receptive field, our formulation has a receptive field of the whole image, which is much larger than segmentation methods. Contextual information and information from other locations in the image can be used to solve problems without visual cues. From a learning perspective, based on our formulation, structural loss can also be used to learn prior information such as the shape and orientation of lanes, as shown in Section 3.2. In this way, problems without visual cues can be handled in our paradigm.

Another important benefit is that this paradigm models lane locations in a row-based manner, which gives us the opportunity to explicitly establish relationships between different rows. The original semantic gap caused by low-level pixelated modeling and high-level lane long-line structure can be bridged.

loss of lane structure

In addition to the classification loss, we also propose two loss functions for modeling the positional relationship of lane points. This facilitates the learning of structural information.

The first one is due to the fact that lanes are continuous, i.e. lane points in adjacent row anchors should be close to each other. In our formulation, the position of the lane is represented by a classification vector. Therefore, continuity is achieved by constraining the distribution of classification vectors over adjacent row anchors. In this way, the similarity loss function is:
$$L_{sim}=\sum _{i=1}^C\sum _{j=1}^{h-1}\Vert P_{i,j,:}-P_{i,j+1,:}{\Vert }_1,$$
of which,$P_{i,j,:}$is $The\; prediction\; of\; j$ row anchors,$\Vert \cdot {\Vert }_1$L $L_1$norm.

Another structural loss function focuses on the shape of the lanes. Generally, most lanes are straight. Even the curves are mostly still straight due to the perspective effect. In this work, we use a second-order difference equation to constrain the shape of the lane, which is zero in the straight case.

To account for shape, the lane position on each row anchor needs to be computed. The intuitive idea is to obtain the position from the classification prediction by finding the peak of the largest response. For any lane index $i$ and the row anchor index$j$，使用$L_{OCi,}$Can be expressed as:
$$L_{OCi,}=\underset{k}{\mathrm{argmax}}{P}_{i,j,k},s.t.k\in [1,w]$$
where$k$ is an integer representing the position index. Note that we are not counting in background grid cells, and the position indexkkThe value range of$k$$1$ to$w$ instead of$w+1$。

However, $The\; argmax$ function is not differentiable and cannot be used with further constraints$.$In addition, in the classification formulation, the classes have no obvious order, making it difficult to establish the relationship between different row anchors. To address this, we propose to use the predicted expectation as an approximation of the location. We use$softmaxx$ function to get the probability of different positions:
$$Pro{b}_{i,j,:}=softmax(P_{i,j,1:w}),$$
where$P_{i,j,1:w}$yes $w-$ dimensional vector,$Pro{b}_{i,j,:}$represents the probability at each location. For the same reason as Eq. 4, background grid cells are not included and the calculation range is only $1$ to$w$ , then the position expectation is:
$$L_{OCI,}=\sum _{k=1}^{w}k\cdot Pro{b}_{i,j,k}$$
Among them $Pro{b}_{i,j,k}$right $i$ lane, the$The\; j-th$ row anchor and the$The\; probability\; of\; k$ positions. The benefits of this targeting method are twofold. The first is that the expectation function is differentiable. The second is that the operation recovers continuous positions with discrete random variables.

According to the above formula, the second-order difference constraint can be written as:
$$L_{shp}=\sum _{i=1}^C\sum _{j=1}^{h-1}\Vert (L_{OCi,}-L_{OCi,j+})-(L_{OCi,j+}-L_{OCi,j+}){\Vert }_1,$$
among them,$j$ forsecondThe position on the$i$$Anchor\; point\; for\; row\; j$ . The reason we use second-order differences instead of first-order differences is that first-order differences are non-zero in most cases. Therefore, the network needs additional parameters to learn the distribution of the first difference of lane positions. In addition, the constraint of the second-order difference is relatively weaker than that of the first-order difference, so when the lane is not straight, the influence of the second-order difference is small. Finally, the overall structural loss can be:
$$L_{str}=L_{sim}+\lambda L_{shp}$$
where λ is the loss factor.

feature aggregation

The above loss design mainly focuses on the internal relationship between lanes. In this section, we propose an auxiliary feature aggregation method that is performed on both global context and local features. An auxiliary segmentation task is proposed to model local features using multi-scale features. We use cross-entropy as an auxiliary segmentation loss. In this way, the overall loss of our method can be written as:
$$L_{total}=L_{cls}+\alpha L_{str}+\beta L_{see},$$
where$L_{see}$is the segmentation loss, $a$和$\beta$ is the loss coefficient. The overall architecture is shown in Figure 4.

It should be noted that our method only uses the auxiliary segmentation task during the training phase, which will be removed during the testing phase. In this way, even if we add additional segmentation tasks, it will not affect the running speed of our method. It is the same as the network without the auxiliary segmentation task.

UFLDv2

On the basis of UFLD, column anchors and row anchors are added to form hybrid anchors, based on which the lane detection task is reformulated as an ordered classification problem to obtain lane coordinates.

We extend the lane representation with row anchors to a hybrid anchor system. Our observation is that row anchor systems may not be suitable for all types of lanes and may cause amplified localization problems. As shown in Figures 2a and 2b, when row anchors are used, the localization accuracy of the side lane is significantly lower than that of the self lane. What if we use column anchors? In Figure 2c, we can see the opposite phenomenon, where the column anchor system is less able to localize the ego channel. We refer to this phenomenon as the amplified localization error problem. This problem makes it difficult for row anchors to locate horizontal (side lane) lanes, and it also makes it difficult for column anchors to locate vertical lanes (ego lanes). Based on the above observations, we propose to use mixed (row and column) anchors to represent different lanes respectively. Specifically, we use row anchors for the self channel and column anchors for the side channel. This can alleviate the problem of positioning error amplification and improve performance.

In a hybrid anchor system, lanes can be represented by coordinates on the anchor system. How to effectively learn these coordinates is another important issue. The most straightforward way is to use regression. In general, regression methods are suitable for local-scale prediction settings and are relatively weak in modeling long-range and global localization . To cope with the global distance prediction, we propose a classification-based lane coordinate learning method , which uses different classes to represent different coordinates. In this work, we further extend the original classification to ordered classification . In ordered classification, adjacent classes have a close ordered relationship , which is different from the original classification. For example, in the ImageNet classification task, class 7 is stingray (a type of fish) and class 8 is rooster. In our work, the classes are ordered (eg, the lane coordinates of class 8 are always spatially to the right of class 7). Another property of ordered classification is that the space of classes is continuous . For example, a non-integer class like class 7.5 makes sense and can be seen as an intermediate class between classes 7 and 8. To achieve ordered classification, we propose two loss functions to jointly model the ordered relationship between classes, including basic classification loss and mathematical expectation loss. Using the order relation and the continuous class space property, we can replace argmax with mathematical expectation to get the predicted continuous class [19]. The expected loss is the continuous class that constrains the predictions to be equal to the ground truth. By simultaneously constraining the base loss and the expected loss, the output has a better ordered relationship, which is beneficial for lane localization.

Anchor-based lane representation

To represent lanes, we introduce row anchors for lane detection, as shown in Figure 3. Lanes are represented by dots on row anchors. However, row-anchor systems may cause amplified localization error problems, as shown in Figure 2. In this way, we further extend the row anchor system to a hybrid anchor system.

The reason for this problem is shown in Figure 5. Assuming that there is no anchor point system, the ideal minimum positioning error is $\epsilon$ , which may be caused by network bias, labeling error, etc. We can see that the error band of the row anchor system is multiplied by$\frac{1}{sin\theta }$. When the angle θ between the channel and the anchor point θ$When\; \theta$ is small, the amplification factor is$\frac{1}{sin\theta }$tends to infinity. For example, when a lane is strictly horizontal, it is not possible to represent that lane with a row anchor system. This problem makes it difficult for row anchors to locate more horizontal lanes (usually side lanes), and likewise it makes it difficult for column anchors to locate more vertical lanes (usually ego lanes). Conversely, when the lane is perpendicular to the anchor point, the error introduced by the anchor point system is minimal ( $\theta$ = 0), which is equal to the ideal positioning error$eh$ .

Based on the above observations, we further propose to use hybrid anchors to represent lanes. For different lane types, different positioning systems are used to reduce the amplified positioning error. Specifically, the rule is: only one kind of anchor can be assigned to a lane, and the more vertical anchor type is chosen for that lane. In practical applications, lane detection datasets such as CULane [2] and TuSimple [53] only label two ego lanes and two side lanes, as shown in Figure 2a. In this way, we use row anchors on ego lanes and column anchors on side lanes, and the hybrid anchor system can alleviate the amplified localization error problem.

In the hybrid anchor system, we can represent the lane as a series of coordinates on the anchor, as shown in Figure 4. $N_{row}$Indicates the number of row anchors, $Ncol$ represents the number of column anchors. For each lane, we first assign the corresponding anchor system with the smallest localization error. Then calculate the straight line intersection of the lane and each anchor point, and record the coordinates of the intersection. If the lane does not intersect between some anchors, the coordinates will be set to -1. Suppose the number of lanes assigned to row anchors is$N_{lane}^r$, the number of lanes assigned to the column anchor is $N_{lane}^c$. The lanes in the image can be represented by a fixed-size target $T$ , where each element is either the coordinates of the lane or -1, and its length is$N_{row}\times N_{lane}^r+N_{col}\times N_{lane}^c$。$T$ can be divided into$T^r$ and$T^{Two\; parts\; of\; c}$ , corresponding to the part on the row anchor and the column anchor, respectively, the size is$N_{row}\times N_{lane}^r$和$N_{col}\times N_{lane}^c$。

Anchor Driven Network Design

With lane representations of mixed anchors, we design the network with the goal of learning a fixed-size target $T_r$and $T_c$and classify. To learn T r T_r with classification$T_r$and $T_c$, we will $T_r$and $T_c$Different coordinates in map to different classes. Suppose $T_r$and $T_c$is normalized ( $T_r$and $T_c$The elements range from 0 to 1 or equal to 1, that is, the "no lane" case), the number of classes is $N_{dim}^r$and $N_{dim}^c$。映射可以写成:
{ T c l s i , j r = ⌊ T i , j r N d i m r ⌋ , T c l s m , n c = ⌊ T m , n c N d i m c ⌋ , s . t . i ∈ { 1 , ⋯   , N r o w } , j ∈ { 1 , ⋯   , N l a n e r } , m ∈ { 1 , ⋯   , N c o l } , n ∈ { 1 , ⋯   , N l a n e c } , \begin{aligned} &\begin{cases} T^r_{cls_i,j}=\lfloor T^r_{i,j}N^r_{dim}\rfloor, \\ T^c_{cls_m,n}=\lfloor T^c_{m,n}N^c_{dim}\rfloor, \end{cases} \\ s.t.\quad &i\in\{1,\cdots,N_{row}\},j\in\{1,\cdots,N^r_{lane}\},\\ &m\in\{1,\cdots,N_{col}\},n\in\{1,\cdots,N^c_{lane}\}, \end{aligned} s.t.​{ Tclsi​,jr​=⌊Ti,jr​Ndimr​⌋,Tclsm​,nc​=⌊Tm,nc​Ndimc​⌋,​i∈{ 1,⋯,Nrow​},j∈{ 1,⋯,Nlaner​},m∈{ 1,⋯,Ncol​},n∈{ 1,⋯,Nlanec​},​
where $T_{cls}^r$和$T_{cls}^c$is the mapping class label of coordinates, $\lfloor \cdot \rfloor$ is rounding operation,$T_{cls\_i,j}^r$为$T_{cls}^r$target $line\; i$ , linejjelements of column$j\; .$In this way, we can convert the learning of coordinates on the mixed anchor point into two dimensions$N_{dim}^r$and $N_{dim}^c$classification problem. For the case of no lane, ie $T_{i,j}^r$or $T_{m,n}^c$equal to -1, we use an additional two-way classification to represent:
T ext _ i , jr = { 1 , if T i , jr ≠ − 1 0 , otherwise , s . t . i ∈ { 1 , ⋯ , N row } , j ∈ { 1 , … , N laner } \begin{align} &T^r_{ext\_i,j}= \begin{cases} 1,\ if T^r_{i,j}\neq -1\\ 0,\ otherwise \end{cases}\ ,\\ &s.t.\quad i\in\{1,\cdots,N_{row}\},j\in\{1,\dots,N^r_{ lane}\}\end{align}​Text_i,jr​={ 1, ifTi,jr​=−10, otherwise​ ,s.t.i∈{ 1,⋯,Nrow​},j∈{ 1,…,Nlaner​}​​
where $T_{ext}^r$is the class label of coordinates, $T_{ext\_i,j}^r$为$T_{ext}^r$ii _row$i$ , jjelements of column$j\; .$column anchor$T_{ext}^c$的存在目标类似:
T e x t _ m , n c = { 1 ,   i f T m , n c ≠ − 1 0 ,   o t h e r w i s e   , s . t . m ∈ { 1 , ⋯   , N c o l } , n ∈ { 1 , … , N l a n e c } . \begin{align} &T^c_{ext\_m,n}= \begin{cases} 1,\ if T^c_{m,n}\neq -1\\ 0,\ otherwise \end{cases}\ ,\\ &s.t.\quad m\in\{1,\cdots,N_{col}\},n\in\{1,\dots,N^c_{lane}\}. \end{align} ​Text_m,nc​={ 1, ifTm,nc​=−10, otherwise​ ,s.t.m∈{ 1,⋯,Ncol​},n∈{ 1,…,Nlanec​}.​​
According to the above derivation, the entire network needs to learn $T_{cls}^r$、$T_{cls}^c$、$T_{ext}^r$和$T_{ext}^c$, which includes both the positioning and existence branches. Suppose the deep feature of the input image is $X$ , then the network can be expressed as:
$$P,E=f\left(flatten\right(X)),$$
where,$Pwa$ EE_$E$ are the location and existence branches respectively,$f$ is the classifier,$flates(\cdot )$ is the flattening operation. $Pwa$ EE_$The\; output\; of\; E$ consists of two parts ($P^r$，$P^c$，$E^r$ and$E^c$ ) composition, corresponding to the row and column anchors, respectively. $P^r$ and$P^{The\; dimensions\; of\; c}$ are$N_{lane}^r\times N_{row}\times N_{dim}^r$和$N_{lane}^c\times N_{col}\times N_{dim}^c$, where $N_{dim}^r$and $N_{dim}^c$is the mapped categorical dimension for row and column anchors. $E^r$ and$E^{The\; dimensions\; of\; c}$ are$N_{lane}^r\times N_{row}\times 2$和$N_{lane}^c\times N_{col}\times 2$。

In the above formula, we directly flatten the deep features of the backbone network and feed them into the classifier. In contrast, traditional classification networks use global average pooling (GAP). We use the flatten operation instead of GAP because we found that for classification-based lane detection networks, spatial information is very important. Using GAP eliminates spatial information, resulting in poor performance.

Ordinal classification loss

As we saw in Equation 1, an important property of the classification network described above is that there is an ordered relationship between classes. In our classification network, adjacent categories are defined as having close and ordered relationships, which is different from traditional classification. To better utilize the prior knowledge of ordered relations, we propose to use base classification loss and expectation loss.

基分类损失定义为:
$$\begin{array}{rl}{L}_{cls}& =\sum _{i=1}^{N_{lane}^r}\sum _{j=1}^{N_{row}}{L}_{CE}(P_{i,j}^r,onehot(T_{cls\_i,j}^r))\\ & +\sum _{m=1}^{N_{lane}^c}\sum _{n=1}^{N_{col}}{L}_{CE}(P_{m,n}^c,onehot(T_{cls\_m,n}^c)),\end{array}$$
份位$L_{CE}(\cdot )$ is the cross entropy loss,$P_{i,j}^r$is the iith assigned to the row anchor$The\; prediction\; of\; the\; i$ lane, corresponding to the$j$ row anchors,$T_{cls\_i,j}^r$is $P_{i,j}^r$corresponding category labels. $P_{m,n}^c$is the mmth assigned to the column anchor$Predictions\; for\; m$ lanes, corresponding to nnth$n$ column anchors,$T_{cls\_m,n}^c$is $P_{m,n}^c$The corresponding classification labels for , $onehot(\cdot )$ is a one-hot encoding function.

Since the categories are ordered, the predicted expectation can be viewed as the average voting result. For convenience, we express expectation as:
$$\{\begin{array}{l}Ex{p}_{i,j}^r={\sum }_{k=1}^{N_{dim}^r}Pro{b}_{i,j}^r[k]\cdot k,\\ Ex{p}_{m,n}^c={\sum }_{l=1}^{N_{dim}^c}Pro{b}_{m,n}^c[l]\cdot l,\end{array}$$

Where [·] represents the indexing operator. The problem is defined as:
$$\{\begin{array}{l}Pro{b}_{i,j}^r=softmax(P_{i,j}^r),\\ Pro{b}_{m,n}^c=softmax(P_{m,n}^c),\end{array}$$
In this way, we can constrain the forecasted expectations to be close to the ground truth. Therefore, we have the following expected loss:
$$\begin{array}{rl}{L}_{exp}& =\sum _{i=1}^{N_{lane}^r}\sum _{j=1}^{N_{row}}{L}_1(Ex{p}_{i,j}^r,T_{cls\_i,j}^r)\\ & +\sum _{m=1}^{N_{lane}^c}\sum _{n=1}^{N_{col}}{L}_1(Ex{p}_{m,n}^c,T_{cls\_m,n}^c)\end{array}$$
where $L1(\cdot )$ is smooth$L1$ loss.

A graphical representation of the expected loss is shown in Figure 7. We can see that the expectation loss can facilitate lane localization by pushing the mathematical expectation of the predicted distribution towards the position of the ground truth.

In addition, the loss Lext of the existence branch is defined as follows:
$$\begin{array}{rl}{L}_{ext}& =\sum _{i=1}^{N_{lane}^r}\sum _{j=1}^{N_{row}}{L}_{CE}(E_{i,j}^r,onehot(T_{ext\_i,j}^c))\\ & +\sum _{m=1}^{N_{lane}^c}\sum _{n=1}^{N_{col}}{L}_{CE}(E_{m,n}^c,onehot(T_{ext\_m,n}^c)).\end{array}$$
Finally, the total loss can be expressed as:
$$L=L_{cls}+\alpha L_{exp}+\beta L_{ext},$$
where α and β are loss coefficients.

network reasoning

In this section, we show how to obtain detection results during inference. Taking the row anchor system as an example, suppose $P_{i,j}^r$Sum $E_{i,j}^r$are the prediction results of the i-th lane and the j-th anchor point, respectively. Then $P_{i,j}^r$Sum $E_{i,j}^r$The lengths are $N_{dim}^r$and 2. The probability of each lane position can be expressed as:
$$Pro{b}_{i,j}^r=softmax(P_{i,j}^r),$$
where the length of Probri,j is N rdim. Then, by taking the mathematical expectation of the predicted distribution, the position of the lane can be obtained. Furthermore, predictions for non-existent lanes are filtered out based on predictions from the presence branch.
$$Lo{c}_{i,j}^r=\{\begin{array}{l}{\sum }_{k=1}^{N_{dim}^r}Pro{b}_{i,j}^r[k]\cdot k,if{E}_{i,j}^r[2]>E_{i,j}^r[1]\\ -1,otherwise\end{array},$$
and finally, the obtained location Loc is scaled to fit the size of the input image. The overall schematic diagram of the network architecture is shown in Figure 6.