Click on " Xiaobai Learning Vision " above , and choose to add " Star " or " Top "
重磅干货，第一时间送达Reposted from: Automated Construction
Automatic recognition of image road surface type based on convolutional neural network
Automatic Pavement Type Recognition for Image-Based Pavement Condition Survey Using Convolutional Neural Network
author:
Guangwei Yang, Ph.D. （Postdoctoral Researcher, School of Civil and Environmental Engineering, Oklahoma State Univ., Stillwater, OK 74078. ORCID: https://orcid .org/0000-0002-0870-2440）
Kelvin C. P. Wang, Ph.D., M.ASCE （2 Regents Professor, School of Civil and Environmental Engineering, Oklahoma State Univ., Stillwater, OK 74078 (corresponding author). Email: [email protected]）
Joshua Qiang Li, Ph.D., M.ASCE （Associate Professor, School of Civil and Environmental Engineering, Oklahoma State Univ., Stillwater, OK 74078.）
Yue Fei, Ph.D. (Algorithm Engineer, Chengdu Guimu Robot, 888 Chenglong Rd., Chengdu, Sichuan 610000, China.)
Yang Liu, Ph.D. （Postdoctoral Researcher, School of Civil and Environmental Engineering, Oklahoma State Univ., Stillwater, OK 74078.）
Kamyar C. Mahboub, Ph.D., M.ASCE （Professor, Dept. of Civil Engineering, Univ. of Kentucky, Lexington, KY 40503. ORCID: https://orcid.org/0000-0003-1668-4585）
Allen A. Zhang, Ph.D. （Senior Researcher, Guangdong Provincial Academy of Building Research, 121 E. Xianlie Rd., Guangzhou 510500, China. ORCID: https:// orcid.org/0000-0002-2565-9894）
期刊：Journal of Computing in Civil Engineering
Original link: 10.1061/(ASCE)CP.1943-5487.0000944.
Q1
What is the engineering question posed by the article?
What is the actual engineering value?
Pavement systems deteriorate over time under the influence of traffic loads and climate, requiring significant annual expenditures to repair and maintain their performance at expected levels. Reliable and accurate pavement condition data play a key role in pavement management systems (PMS), manual investigations can have potential safety issues, require traffic control, are time consuming and the results are subject to subjectivity, which can lead to inconsistent pavement condition data from year to year .
Therefore, a survey technique based on digital imaging captures road surface images at highway speeds and stores them on an electronic medium, and methods for performing interpretation of road surface conditions have been intensively studied. Automated and semi-automated technologies have gained wide acceptance in the field of road condition data collection due to their advantages in terms of safety and efficiency, data consistency and repeatability, and high-resolution road surface images with full lane coverage.
Q2
What is the scientific question raised by the article?
What new academic contributions?
Image-based systems are becoming popular for collecting pavement condition data for pavement management activities, and pavement engineers define various distress categories based on pavement type. However, current software solutions have limitations in automatically correctly identifying the type of road surface from the collected images.
In this paper, we propose a Convolutional Neural Network (CNN) based pavement recognition system, PvmtTPNet, with acceptable consistency, accuracy and efficient automatic recognition of pavement types.
1. By analyzing the sound profile and texture of the near field using statistical learning methods, different types of road surfaces were identified.
2. Pavement images at 1 mm resolution on routes with different pavement types under different conditions in Oklahoma using the state-of-the-art PaveVision3D system (Wang et al. 2015). A total of 80% of the prepared images are randomly selected for training the proposed network, and the remaining 20% of the images are used for testing.
3. Apply the obtained network to determine the road surface type for two additional data collection images in 2019 to evaluate the performance.
Q3
What is the technical route proposed in the article?
What are the improvements and innovations?
①Training data
This study considered three pavement types commonly assessed and measured in PMS: asphalt concrete pavement (AC), joint plain concrete pavement (JPCP), and continuous reinforced concrete pavement (CRCP). A total of 21,000 two-dimensional (2D) images were collected covering 84,000 m (52.20 miles) of long road surface slices. 80% of the prepared images are randomly selected for training the proposed network, and the remaining 20% of the images are used for testing. During the training process, the prepared 2D images are reduced to 475×512 2D images to improve computational efficiency. Figure 1 is an image sample of the pre-prepared dataset.
Sample images of the dataset prepared in Fig. 1: (a) new; (b) with cracks; (c) with sealed cracks; (d) with repairs; (e) with cracks; (f) with PCC repairs; ( g ) with outlet; (h) with AC patch; (i) with DBR; (j) with DBR and patch; (k) with crack; (l) with groove.
② Web development
Figure 2 shows the architecture of PvmtTPNet. PvmtTPNet consists of six layers: three convolutional layers, two fully connected layers, and an output layer. The input of PvmtTPNet is a prepared two-dimensional road surface image, and the output layer calculates the probability distribution of the predicted road surface type. In each convolutional layer, 8 kernels of size 13×13 are used to extract features of the input image, such as edges and shapes. For these two fully connected layers, we implement 32 nodes and 16 nodes respectively to preserve the most important features of road surface images.
Figure 2 Flow chart of optimizing HNN using GA
③Training skills
During network training, a combination of different techniques is used to tune hyperparameters within PvmtTPNet based on prepared 2D images. The parameters of the network are gradually adjusted to reduce the error between the output score and the expected score pattern, so as to reduce the training loss and improve the training accuracy (LeCune et al. 2015). After extensive training, PvmtTPNet is able to predict the road type for a given 2D image based on a score vector, where the highest score across all categories will correspond to the road type. Table 1 summarizes the tuning hyperparameters of PvmtTPNet, with a total of 992,979. Table 1 is a summary of the training parameters.
Table 1 Summary of training parameters
④ Training results
The classification accuracy and cross-entropy loss for network training and testing are shown in Fig. 3. As the number of training epochs increases, the classification accuracy increases and the cross-entropy loss decreases. The training of PvmtTPNet on the prepared dataset for 100 epochs takes 28 hours to complete on an NVIDIA titan VGPU card. With the chosen combination of training techniques, the classification accuracy on the test data is still close to that on the training data, which indicates that there is little overfitting problem in this network. In particular, PvmtTPNet achieves the highest test accuracy of 98.48%, which is observed at epoch 96. Meanwhile, the cross-entropy losses of training data and testing data are 0.0067 and 0.054, respectively. Therefore, the parameters derived at stage 96 are considered to be the optimal parameters for PvmtTPNet. The classification accuracy of the training data reaches 99.83% in the optimal period.
Figure 3 Summary of classification accuracy and cross-entropy loss
Based on the above, this paper has the following innovations:
1. This study develops a deep learning (DL)-based network, called PvmtTPNet, that can automatically identify road surface types from images to facilitate fully automated road surface condition surveys. PvmtTPNet implements a convolutional neural network based architecture to learn features from images of the road category.
2. Using rectified linear units (ReLUs) as the activation function of the convolutional layer and the fully connected layer can be trained quickly and effectively, and has become the default activation function of modern deep learning neural networks.
Q4
How is the article validated and resolved?
In order to evaluate the performance of the obtained PvmtTPNet during the optimal period of pavement type recognition, additional studies were performed on Site 1 (near Oklahoma City) and 2 (near I-540 in Fort Smith, Arkansas) by the PaveVision3D system in 2019. Two field data collections. The two data collection paths are shown in Figure 4.
Figure 4. Data collection for model evaluation:
(a)Site1-I-35；(b)Site2-I-540
Table 2 summarizes the detailed quantities of actual and predicted pavement types collected by PvmtTPNet for these two data collections and provides a confusion matrix for each site.
Table 2 Confusion matrix in online evaluation
In each confusion matrix, numbers along the diagonal represent correct predictions, while other numbers represent accurate predictions for each pavement class. As shown in Table 2, PvmtTPNet obtained 5760 correct predictions from the 6311 images prepared by Site1, with an accuracy rate of 91.27%. For Site2, PvmtTPNet made 3,439 correct predictions and achieved an accuracy rate of 96.66%. In addition, it takes 16.33 min and 4.59 min for PvmtTPNet to predict images from Site 1 and Site 2 using one NVIDIA titan VGPU card. Therefore, the average processing time per image was 155,212 μs for site 1 and 77,452 μs for site 2. If the field data collection speed is 96.56km/h (60mi/h), it takes 18.55min and 10.46min to complete the survey of these two sites. Therefore, the processing time was less than the data acquisition time (16.33 < 18.55 for site 1 and 4.59 < 10.46 for site 2). PvmtTPNet clearly has the potential to use state-of-the-art GPUs to predict road surface types from real-time collected 2D images, which will be several times faster than the GPUs used in the study.
Q5
What are the advantages and disadvantages of the article?
Logical structure : The outline of this article is presented below:
1. Introduction
It shows that there are defects in obtaining road condition data manually. Therefore, it is necessary to study the automatic recognition of road surface types in images based on convolutional neural networks.
2. Data preparation
All pavement image sources used in this study are introduced, as well as the objects on which PvmtTPNet is trained in this paper.
3. Network Development
In this study, a deep learning (DL)-based network, called PvmtTPNet, is developed to automatically identify road surface types from images to facilitate fully automated road condition surveys. PvmtTPNet implements a convolutional neural network based architecture to learn features from images of the road category.
3.1 Methodology
Introduce the methodology of this study to train the proposed PvmtTPNet with a CNN architecture.
3.2 Network Architecture
Introduce the network architecture of PvmtTPNet.
3.3 Training Techniques
Introduce the data source and training method of PvmtTPNet network training.
3.4 Training Results
Show classification accuracy and cross-entropy loss for network training and testing. With a combination of choices of training techniques, the classification accuracy of the test data is still close to that of the training data, indicating that there is little overfitting problem in this network.
4. Network Evaluation
The PaveVision3D system collects data from Oklahoma City and I-540 near Fort Smith, Arkansas, and evaluates its performance in the best period of road surface type recognition through the results of PvmtTPNet in road surface type recognition.
5. Discussion
Currently, adding event markers on bridges during data collection is a common method for excluding bridge sections from collected image data. However, this is an estimate of the field workforce, which may produce an inaccurate record given the high rate of data collection. Therefore, in the next stage of work, the training effect of PvmtTPTet on bridge deck images will be judged based on the obtained images. But there are several limitations: First, there are not enough images collected from the bridge deck. It is well known that DL training requires a large amount of training data to achieve the desired performance. Second, 2D images of bridge decks do not always contain the discriminative features learned by this network.
6. Conclusions
In this study, a convolutional neural network-based DL network, named PvmtTPNet, was trained to recognize road surface types in humans. In 2018, a training database was compiled using the PaveVision3D system to investigate three types of asphalt concrete pavement, connected ordinary concrete pavement, and continuous reinforced concrete pavement with different conditions and pressures in Oklahoma. In the end, a total of 21,000 2D road surface images were produced, while about 7,000 images were made for each of the three road surface types. Each 2D image covers a pavement section approximately 4 meters wide and 4 meters long. With the chosen training technique, the network was successfully trained without overfitting issues. In the optimal period, the prediction accuracy of the network on the training and test images of road surface type recognition reaches 99.85% and 98.37%, respectively.
It should be noted that the image of the bridge deck is not included as a pavement type in PvmtTPNet. Therefore, future research hopes to identify bridges using more datasets and possibly newer DL methods. Finally, PvmtTPNet needs to be improved to produce more accurate predictions for images on rigid road surfaces. The ultimate goal is to achieve near 100 percent accuracy for automatic and high-speed identification of asphalt and concrete pavement types, as well as other surface types such as bridge decks and composites.
From the above content, it can be seen that this paper mainly adopts a vertical structure, and introduces data processing, neural network development, evaluation of network training results, and further development of automatic recognition of road surface types in convolutional neural network images in the order of research development.
Research method : During the research process of this paper, the method of evaluating the performance of the convolutional neural network PvmtTPNet on the collected road surface types is very detailed, and the collection effect of PvmtTPNet in predicting these two types of data is verified from multiple perspectives.
Figure 5 PvmtTPNet in the prediction data accuracy evaluation formula
Graphical form : The graphic form of this article is concise and clear, without using complicated graphical tables, but it intuitively shows the experimental results.
Summary of classification accuracy and cross-entropy loss
Confusion Matrix for Web Assessment
Text expression :
Table 2 summarizes the detailed quantities of actual and predicted pavement types collected by PvmtTPNet for these two data sets, and provides a confusion matrix for each site.
In the evaluation of image accuracy, the confusion matrix is mainly used to compare the classification results and the actual measured values, and the accuracy of the classification results can be displayed in a confusion matrix. The confusion matrix is calculated by comparing the position and class of each observed pixel with the corresponding position and class in the classified image. Each column of the confusion matrix represents the actual measured information, and the value in each column is equal to the number of actual measured pixels corresponding to the corresponding category in the classification image; each row of the confusion matrix represents the classification information of the remote sensing data, and each row The value in is equal to the number of remote sensing classification pixels in the corresponding category of measured pixels.
Figure 6 The author uses the context of the confusion matrix in the article
Q6
How does the article inspire your own research?
This paper mainly studies the automatic recognition of road image types through convolutional neural networks. After validating PvmtTPNet with sufficient samples, it is extended to explore that images of bridge decks are not included as a type of road surface in PvmtTPNet, hoping to use more datasets and possibly updated DL methods to identify bridges. And PvmtTPNet is improved to produce more accurate predictions for images on rigid road surfaces. Applications are extended for automatic and high-speed identification of asphalt and concrete pavement types and composite materials.
When we do scientific research, we should draw inferences from one instance and extend our own research results. We should not only confine ourselves to the current professional framework, but actively explore more possibilities.
This article is only for academic sharing. If there is any infringement, please contact to delete the article.
下载1：OpenCV-Contrib扩展模块中文版教程
在「小白学视觉」公众号后台回复：扩展模块中文教程，即可下载全网第一份OpenCV扩展模块教程中文版，涵盖扩展模块安装、SFM算法、立体视觉、目标跟踪、生物视觉、超分辨率处理等二十多章内容。
下载2：Python视觉实战项目52讲
在「小白学视觉」公众号后台回复：Python视觉实战项目，即可下载包括图像分割、口罩检测、车道线检测、车辆计数、添加眼线、车牌识别、字符识别、情绪检测、文本内容提取、面部识别等31个视觉实战项目，助力快速学校计算机视觉。
下载3：OpenCV实战项目20讲
在「小白学视觉」公众号后台回复：OpenCV实战项目20讲，即可下载含有20个基于OpenCV实现20个实战项目，实现OpenCV学习进阶。
交流群
欢迎加入公众号读者群一起和同行交流，目前有SLAM、三维视觉、传感器、自动驾驶、计算摄影、检测、分割、识别、医学影像、GAN、算法竞赛等微信群（以后会逐渐细分），请扫描下面微信号加群，备注：”昵称+学校/公司+研究方向“，例如：”张三 + 上海交大 + 视觉SLAM“。请按照格式备注，否则不予通过。添加成功后会根据研究方向邀请进入相关微信群。请勿在群内发送广告，否则会请出群，谢谢理解~