Visual lane line tracking and prediction for autonomous vehicles
Summary
We propose a visualization method for autonomous vehicles to track the position of lane markings on horizontal roads. Our method is based on predictive filtering. The prediction step estimates the expected lane marking position in each new image frame. It is alsobased on information produced by the car's kinematic model and embedded odometry sensors. Experimental results obtained using properly prepared test vehicles show that under certain conditions, such asoscillations and lane changes, the prediction step can significantly reduce tracking errors . Therefore, we believe that the application of our method to image-based control of autonomous vehicles can improve system performance.
1 Introduction
The car itself is undoubtedly one of the biggest challenges of modern automation. On the one hand, car pilots of airplanes and ships are well known and widely used. For cars and trucks, on the other hand, the technology is still under development due to greater complexity. The goal is for a car to autonomously drive from address A to address B as safely (or more safely) than an experienced human driver. In this case, the user is just a passenger.
For cars, a basic requirement for autonomous driving systems is to be able to keep the vehicle within the road and drive in the center of the lane. This requires knowledge of the surrounding environment, which is obtained through sensors installed on the car, such as cameras, laser scanning, GPS and inertial units.
Projects such as (NASHMAN, 1992), (SCHNEIDERMAN, 1994), (POMERLEAU, 1996), (BROGGI, 1999) and (LU, 2002) rely only on grayscale images generated by cameras for road detection. position and estimated curvature. In (CHRISMAN, 1998) and (SALES, 2010), color images were used to classify areas belonging to roads. This idea has also been applied in (DAHLKAMP, 2006) to extend the range of the system beyond the range of the laser rangefinder (LRF) sensors used on their autonomous vehicles.
Stereo vision systems allow the calculation of the distance of objects on the scene. A binocular vision system (stereo vision) was used for obstacle detection in (BROGGI, 2010) and (LIMA, 2010). However, this system has the disadvantage of high computational cost.
Projects like (GUIZZO, 2011) and (Autonomous Driving Laboratory, 2012) combine LRF sensors with cameras, GPS, IMU and map information to bring autonomous driving research to unprecedented levels. Both projects successfully accomplished autonomous navigation tasks in real urban environments.
In Brazil, research in this field is beginning to develop, and we highlight a few: Driving for You, UNIFEI (VEERMAS, 2010), CADU, UFMG (LIMA, 2010), VERO, CTI /Campinas (VERO, 2012), EESC/USP (SENA, 2012),, 2010), developed by ICMC/USP. Another group at our university (UFES) is also developing an autonomous vehicle project. They buy a car with the major hardware components already installed, and their main focus is on the software side. Their main goal is to understand how the human brain uses images to interpret the world (LCAD, 2013). On the other hand, the main goal of our project is to build an autonomous car with a simple and low computational cost solution. To achieve this, we are adapting ordinary cars by installing actuators and sensors. Our cars are already able to drive autonomously on real roads under controlled conditions (VIVACQUA et al., 2012). CARINA (CARINA
Our system consists of a laptop connected to a USB camera and an interface board. The USB camera is mounted on the front windshield and pointed towards the road. The interface board connects the computer to the automotive encoder and to the stepper motors responsible for drive wheel positioning.
The visual information produced by computational vision systems is affected by noise or errors caused by occlusions, reflections, shadows, etc. To minimize the impact of these issues, the raw images are subjected to a filtering process to produce more reliable information. This information is the basis for the so-called model state and is fed back into the vision algorithm to define the image region of interest (ROI) (see Figure 1). In (VIVACQUA et al., 2012) this information is also used by the controller to keep the autonomous vehicle centered on the road.
Figure 1: Visual information flow used to generate model states
Some visual lane tracking methods (SCHNEIDERMAN, 1994), (AUFRERE, 2000), (LU, 2002) and (CHOI, 2012). They all use some kind of filtering process to update their model state (parameters that describe lane markings), but none of them use predictions. In order to achieve prediction, the vehicle's displacement information must be accessible and reliable . We have made the necessary adjustments to our self-driving car, and in this paper we propose a visualization method based on predictive filtering to track the position of horizontal road lane markings< /span>. The following sections will explain our system and discuss some experimental results. estimates the desired lane marking position in each new image frame to minimize the tracking error. The prediction step
2. Frame of reference
Before proceeding with the prediction step, it is necessary to change the system reference frame from the camera's reference frame to the car's reference frame, taking into account that the camera is at the origin (self-reference frame - OR). Horizontal lane markings detected by vision systems used in autonomous cars (VIVACQUA, 2012) are mapped by inverse perspective projection (Fig. 2) as if the image were seen from the sky, assuming the road is a perfect plane ( Aerial View).
Figure 2: Frame of reference. (A) Camera reference system; (B) Car reference system
In a self-reference frame, the origin is the point directly below the camera in the ground plane, the Z-axis corresponds to the displacement direction of the car (when moving forward), and the X-axis is the lateral direction. Figure 2(B) shows two wires (left and right) corresponding to the information detected in the image of Figure 2(A). The basic idea of the prediction step is that can predict the position of the car in the next image frame based on the kinematic model, and Position on the actual frame.
3. Car kinematics model
The car kinematics model can be approximated by the bicycle model shown in Figure 3. In the figure: Pr is the center of the circle describing the vehicle trajectory, Pc is the camera position (or origin) in the vehicle, R is the radius of curvature of the trajectory described by Pc, Lt is the distance between the front and rear axles of the vehicle, and Lr is the distance between the origin and the rear axle. distance, α is the front wheel angle.
Figure 3: Car kinematics model
Using the car kinematics model (1), based on its parameters and the front wheel angle (α), we can calculate the vehicle trajectory in the world reference frame (WR).
（1）
Where, x, z, θ are the position and direction of the vehicle in WR, and v is the linear velocity of the vehicle. It is worth noting that the origin of the world reference frame WR is fixed on the ground, not fixed on the car. Therefore, when the car moves, point Pc describes the point-line trajectory, and the position of the fixed object in the world remains unchanged.
4. Location prediction
In the OR reference frame, point Pc coincides with the origin, and fixed objects in the world rotate Pr, but in the opposite direction. Figure 4 shows the initial position of a fixed object in the world as seen in the or reference system. Based on the information about the initial position and the trajectory defined by the kinematic model, the future position can be estimated.
Figure 4: Object position prediction: (a) turn right; (b) turn left; (c) go straight.
The center of rotation (Pr), the radius of curvature (R) and the angular displacement (Δθ) of the camera trajectory are calculated using (2), (3) and (4).
Distance in meters (obtained via an encoder located on the front axle) and front wheel rotation angle (α) (obtained via another encoder mounted on the drive axle):
（2）
（3）
（4）
Where, ΔL is the displacement distance.
5. Parabolic rotation
To estimate the new position of any object, one only needs to know the center of rotation (Pr), the angle of rotation (Δθ), and apply a 2D rotation operation. In the specific case of autonomous vehicles described in (VIVACQUA, 2012), the system was tested with the tracking objects being two road lane markings that constitute a road model. Each mark is represented by a quadratic polynomial.
（5）
Given a point (X, Z) of the rotated parabola, we can find the corresponding point (X, Z) in the original parabola through the reverse rotation operation, given by
（6）
（7）
where Px and Pz are the coordinates of the rotation center. Substituting (6) and (7) in (5), we obtain the general form of the parabola of rotation:
（8）
This form is incompatible with the original data structures used in the software and does not allow explicit calculation of X given Z, which is computationally undesirable. To solve this problem, we propose a method to find a good approximation of the parabola of rotation that maintains the default form given in (5).
Considering the parabola equation of rotation
（9）
The problem we need to solve is to find the coefficients a2r, a1r and a0r. The general idea to solve this problem is: find the center point M[z0..z1] of the original parabola x (z) in the interval; apply Δθ to rotate around the Pr point (thus making M become Mr); consider that Mr belongs to the new parabola xr (z); and consider that the tangent direction of this point is subtracted by the rotation angle Δθ.
The inclination angle of the tangent line at point M is given by
. Using the properties of tangent differences, we have:
. Since the curvature of the rotating parabola does not change, we have a2r = a2. The coefficient a1r can be calculated using 
, and the coefficient a0r can be calculated using 
. By doing this, we obtain the parameters of the parabola of rotation in the OR reference frame.
Figure 5: Approximation of the parabola of rotation
6.Filtering
We consider that the lane marking position will not change much in consecutive frames, and a large amount of noise will cause rapid changes in the image (such as reflection or occlusion). For this purpose, we chose to use a first-order low-pass filter, taking into account its impact on the system behavior, we defined this time constant in the experiment: if this value is too low, the filtering has little attenuation effect on sporadic error measurements. On the other hand, if it is too high, this will delay response and cause difficulties for the model, such as the slow changes in road shape that occur before and after corners. After testing with videos captured by a vehicle-mounted camera at 80km/h, we found that a value of τ = 3 yields a good compromise between robustness and responsiveness.
Equation (10) represents the vector Wi representing the state model. Equation (11) is the expression of the low-pass filter that updates the model state from the metrics generated by the visual system (Ui).
（10）
（11）
7.Experimental results
To measure the performance of the prediction system, we conducted some experiments using a suitably adapted test vehicle to acquire road images synchronized with mileage measurements. The vehicle was driven under three curvature conditions, as shown in Figure 6: (A) turning left; (B) in a straight line; and (C) turning right. In each curvature condition, the vehicle proceeds in three different modes with respect to the oscillation level of the trajectory (Figures 6 (D) to 6 (F)): no oscillation (L0), the vehicle travels along the road parallel to the wire; low oscillation level (L1), the vehicle periodically crosses the center line; high oscillation (L2), the vehicle also crosses the center line, but with a more obvious deviation.
Figure 6: Experimental conditions: (A) turn left, (B) go straight, (C) turn right, (D, E, F) corresponding routes
Introducing oscillations in the trajectory forces the vehicle to move non-parallel to the road line so that predictions can demonstrate their ability to maintain model states close to instantaneous measurements.
The test was conducted on a 100-meter path at approximately 40 km/h, with data (images and odometry) collected every 40 ms. Figure 7 shows the registered drive wheel position for each trajectory in Figure 6.
Figure 7: Drive wheel position data obtained during the test
The data collected is analyzed in the laboratory and used to evaluate system performance (with and without the use of predictions). The performance index used is the absolute horizontal average error (AHAE), calculated as the difference between the curve defined by the model and the curve defined by the instantaneous measurements.
（12）
8. Discussion
Figure 8 shows the frame sequence obtained from the video obtained during the test, which corresponds to the case of the L2 trajectory shown in Figure 6(D). This sequence corresponds to the vehicle's extension away from the centerline, which can be moved from the yellow marked successive displacements to the left.
Figure 8: Frame sequence showing the tracking process with and without prediction.
Figure 8 clearly shows that the model using filtering and prediction (dashed white line) is closer to the metric value (continuous red line) than the model using only filtering and prediction (dashed black line). This illustrates the benefit of our approach.
Another way to get a closer approximation between the perceptual model state and the visual measurements is through the plot in Figure 9, which shows the parameters a0 (lane offset) and a1 (lane) of the filter model and the polynomial with the predicted filter model direction). The curvature parameter (a2) is not given since it is virtually constant in the stretch considered.
Figure 9: Model parameter performance of filter-only and filter-predicted
Figure 10: Cumulative error in oscillation level 2
In Figure 9, we can notice the ability of low-pass filtering to remove high-frequency noise, which results in a delayed smooth curve (without prediction case). By using prediction, undesired delays (forecasting) can be reduced. This reduction is because the prediction compensates for the effect of car motion on the captured image. Figure 10 shows the normalized cumulative error (NAE) of the exponential AHAE under test conditions with higher oscillation (L2), where the cumulative error without prediction is considered to be 1. Calculated using (13), where n is the total number of frames used in the experiment.
（13）
In all test cases, the cumulative error increase rate with prediction is smaller than that without prediction (lower than 0,51).
Figure 11 summarizes the final NAE values obtained under all test conditions, see Figure 11 for details. We can notice that the value of NAE decreases when the car travels on a trajectory with higher oscillation levels. This is because when oscillations are present, the lane markings are displaced laterally within the camera's field of view, and the use of predictions compensates for this effect, bringing the model closer to the real measurements while maintaining the robustness given by the filtering process. We believe that theprediction step will improve the performance of autonomous vehicles when driving under similar conditions.