Detecting bird's nests on power lines, using a detection method that combines SSD and HSV color space filters - there is one less picture in the paper, which is less interesting.

Detection of Bird Nests on Power Line Patrol Using Single Shot Detector

Abstract

• On the power towerbird's nestThe existence poses a threat to the safety and stability of transmission lines. In recent years, using drones to detect bird nests on transmission lines has become one of the important tasks of power inspection. The migration of image processing methods from computer vision to power image recognition is increasingly becoming a trend. This paper proposes a detection method that combines a single-lens detector with an HSV color space filter, using image features with a large color span under different illumination angles to identify bird's nests . The fine-tuned single-shot detector network is used to identify bird's nests, and the detection results are cropped into sub-images. The subimage is then filtered using a selector based on the HSV color space, which does not contain nest objects that can be removed by pixel percentage . Experimental results show that this method can accurately detect bird's nests in test transmission line detection images, with an accuracy of 98.23%. Compared with other single traditional methods, the bird's nest detection method proposed in this paper that combines deep learning and HSV color space filters greatly improves the detection accuracy .

• 论文地址：Detection of Bird Nests on Power Line Patrol Using Single Shot Detector | IEEE Conference Publication | IEEE Xplore

INTRODUCTION

• In recent years, with the improvement of the ecological environment and the strengthening of the protection of wild birds, the number of various types of birds has increased, and bird activities have become more frequent. The branches of the bird's nest are easily blown away by strong winds, causing the wires in the bird's nest to fall on the power lines, causing a ground trip accident. The falling bird droppings may form conductive channels between the insulators, causing the insulators to flash . Among the total number of line failures caused by bird activities every year, line failures rank third after lightning damage and external damage , directly affecting the safety and stability of transmission lines. At present, the inspection of bird's nests mainly relies on manual inspection or manual screening of aerial images, which requires a large workload and low efficiency. Therefore, automatic detection of aerial images during transmission line inspection operations can reduce the burden on power workers and has important practical significance.

• Nowadays, the use of drones to inspect overhead transmission lines has been gradually applied across the country. Drone inspections mainly take a large number of key components of transmission lines and then detect key components in these images. There are various studies on the identification and detection of key components of transmission lines. The research results on pole and tower identification are as follows.

• Liu proposed the Haar feature and cascaded AdaBoost impact hammer identification algorithm, integrating machine learning algorithms into hammer detection, and achieved good results in identification.

• Peng took advantage of the characteristics of high density and large height difference of poles and towers in the laser point cloud, and used regular laser point clouds to locate the poles. The calculation amount was large in the background of large scale and wide field of view, and the effect was not good due to misjudgment in multi-pole tower recognition. good.

• In transmission line identification, Tang proposed a method that uses adaptive threshold Hough transform to detect candidate lines in images, and then distinguishes power lines from candidate lines through fuzzy clustering algorithm. This method is sensitive to interruptions and is sensitive to transmission line images. High quality requirements.

• Du used Hou's heuristic knowledge set to determine the prior and posterior probabilities of the traditional Bayesian classifier, and proposed the idea of ​​improving the Bayesian classifier, applying machine learning ideas to the identification of transmission lines, but Recognition results are poor due to factors such as computing power.

• Luqman Maraaba integrated the feedforward neural network into the Bird's Nest model, first used an edge segmentation algorithm to filter the background, and then trained a multi-layer feedforward neural network to predict the degree of insulator pollution, developing a tool that can accurately estimate the degree of insulator pollution.

• With the rapid development of deep learning in the field of image target recognition in recent years, target detection algorithms based on neutral networks such as SPP-NET, SSD, YOLOv3, RCNN, Faster R-CNN and other neutral networks have achieved great results in the detection of key components of transmission lines. score.

• For example, Wang uses the Faster-RCNN neural network to detect multiple types of components including insulators, impact hammers, and spacers. Its performance is better than traditional manual algorithms, but it is difficult to achieve better results through individual detection.

• Zenan Ling divided the insulator self-explosion positioning problem into two parts, namely the target detection problem based on fastrcnn and the pixel classification problem based on U-net, which is a novel method.

• Michael Gerke trained a line-aware object detection network that learns different features from different images.

• The above research can be divided into two ideas: First, the traditional method of processing power images through modeling, thresholding, clustering and other methods is limited to the identification of single components. The identification algorithm is often cumbersome and has low accuracy. Secondly, the deep image recognition method aims to find the most suitable function and uses the auxiliary tools of the convolutional network to fit and train a large number of power images. The effect is better, but the larger the hardware, the larger the data required, which often requires A longer training period can achieve more accurate recognition results . Considering the harmfulness of bird's nests and balancing the recognition accuracy and speed of existing algorithms, it is necessary to conduct further research on the detection of bird's nests.

• In response to the above problems, a bird's nest detection method combining Single Shot Detector and hsv color space is proposed. First, the Single Shot Detector target detection network trained with fine-tuning is used to detect the power patrol image of the bird's nest, intercept the sub-image area of ​​the test result, and then set the color space according to the color range of the bird's nest to filter these areas, and then filter these areas according to the pixels of the bird's nest area. The sub-images are re-screened based on fullness, and finally the detection of the bird's nest image is completed .

DETECTION PRINCIPLE OF BIRD NESTS

• The images taken by drones have high resolution, but the bird's nest area is small, which increases the challenge of bird's nest detection. Due to different shooting angles, focal lengths, and light angles, images have blurred outlines and different scales. In order to improve the accuracy of bird's nest detection in power patrols, a bird's nest detection method based on deep convolutional neural network and color space is proposed. The detection principle is shown in the figure below. First, the pre-trained model needs to be further fine-tuned on the Bird's Nest dataset through the Single Detector Shot convolution network, that is, the remaining convolutional layers of the pre-trained model are frozen and the remaining convolutional layers are trained. This is called fine-tuning; Secondly, the fine-tuned Single Shot Detector detection network is used to initially detect the power patrol bird's nest image; finally, the corresponding detection area is intercepted from the original image according to the prediction frame, and the detection area is filtered through a filter based on the hsv color space. Finally, Complete bird's nest detection in power images .

  • 

Detection Algorithm of Single Shot Detector

• With the rise of deep learning technology, the ability of neural networks based on convolutional structures to classify images has gradually received attention. In recent years, mainstream convolutional neural networks such as Faster-RCNN and YOLO have been widely used in image classification and recognition. Faster-RCNN, as a classic target detection network, uses a two-stage detection strategy and a shared convolution layer strategy, so its detection accuracy is high, but the detection speed is slow. The YOLO target detection network developed later adopts an end-to-end detection method, which improves the speed of image detection, but sacrifices a certain degree of accuracy. The Single Shot Detector network inherits the detection speed of YOLO and the accuracy of Faster R-CNN. It meets the requirements of detection accuracy and real-time performance through multi-layer multi-scale detection and the area detection mechanism inherited from Anchor Box. As shown in the figure above, the network consists of a VGG16 network and a multi-scale feature extraction network. The network consists of VGG16 network and multi-scale feature extraction network. Single Shot Detector converts the last two fully connected layers of VGG16 into convolutional layers, and then adds 4 convolutional layers for multi-scale bird detection .

• Bird's nests in images usually have different scales. Conventional methods usually detect on one layer of feature maps, and multi-scale target detection does not work well. Low-level feature maps have detailed information, while high-level feature maps have high-level semantic information. SSD combines both nest detection.

• The output of SSD is divided into six stages, each stage outputs a feature map, and then performs boundary regression and classification. The Single Shot Detector network uses the first five layers of the VGG16 convolutional network as the first-stage output layer, and then converts the two fully connected layers fc6 and fc7 in VGG16 into two convolutional layers of Conv6 and Conv7, respectively, as the second layer of the network. and the third stage.

• In addition, the single detector network continues to add Conv8, Conv9, Conv10 and Conv11 to extract higher-level semantic information. In each stage operation, the network contains multiple convolutional layer operations, each operation is essentially a small convolution. The feature maps of these additional layers will produce the position offset and confidence of the object through small convolution operations. In target detection, large convolution kernels tend to capture large targets, while small receptive fields can locate small targets. Therefore, the top feature map is likely to miss details of the object. Add Inception blocks to replace the last few layers to enhance detection .

• Faster R-CNN proposes an Anchor-based region selection mechanism, Anchor Box, which generates multiple rectangular boxes on the feature map of the convolutional network based on a given Scale and Radio. However, Faster RCNN only uses the anchor mechanism in the last layer of convolution.

• Single Shot Detector uses the anchor mechanism in Faster R-CNN to generate Default Boxes based on fixed Scale and Radio, as shown in the figure below, and applies it to multiple different feature maps, making the network receptive Field on different layers Different, the detected object scales are more diverse.

  • 

• The loss of Single Shot Detector includes classification loss and regression loss.

  • L ( x , c , l , g ) = 1 N ( L c o n f ( x , c ) + α L l o c ( x , l , g ) ) L l o c ( x , l , g ) = ∑ i ∈ P o s N ∑ m ∈ { x , y , w , h } x i j k s m o o t h L 1 ( l i m − g ^ j m ) L c o n f ( x , c ) = − ∑ i ∈ P O s N x i j p l o g ( c ^ i p ) − ∑ i ∈ N e g l o g ( c ^ i 0 ) L(x,c,l,g)=\frac{1}{N}(L_{conf}(x,c)+\alpha L_{loc}(x,l,g))\\ L_{loc}(x,l,g)=\sum^N_{i\in Pos}\sum_{m\in \{x,y,w,h\}}x^k_{ij} smooth_{L1}(l^m_i-\hat g_j^m)\\ L_{conf}(x,c)=-\sum^N_{i\in POs}x^p_{ij}log(\hat c^p_i)-\sum_{i\in Neg}log(\hat c^0_i) L(x,c,l,g)=N1​(Lconf​(x,c)+αLloc​(x,l,g))Lloc​(x,l,g)=i∈Pos∑N​m∈{ x,y,w,h}∑​xijk​smoothL 1(lim​−g^​jm​)Lconf​(x,c)=−i∈POs∑N​xijp​log(c^ip​)−i∈Neg∑​log(c^i0​)

  • In the formula, N is the number of positive samples, that is. The default number of successful boxes, when N = 0, the entire network loss is set to 0. Alpha α is a hyperparameter that controls the ratio of classification loss and regression loss. In the implementation, Single Shot Detector sets it to 1 through cross-validation.

• Smooth L1 is used in Regression, similar to Fast R-CNN. L1 represents the offset between the predicted box and the default box, that is, the four outputs of the network, and gj represents the offset between the ground truth box and the default box. This function brings the predicted value closer to the true value by aligning the difference between the predicted value and the default box and the difference between the actual value and the default box.

Detection Algorithm of Color Space on Bird Nests

• Color space is a representation of color coordinates and color subspaces. There are many color spaces, the commonly used ones are RGB, CMY, HSV, HSI, etc. It is proposed that the HSV color space is easy to digitize. This paper first transforms the color space of the bird's nest candidate area, and uses the HSV color space to quantitatively extract the color characteristics of the bird's nest.

• The interval selector is as follows: first, convert the Power Patrol bird's nest image candidate area from RGB color space to HSV space, normalize the H component in HSV to 0-180, and normalize the S and V components to 0-240. Secondly, according to the difference in color and subjective perception range, the H (hue) component is divided into 9 sub-intervals, and the S (saturation) component and V (lightness) component are divided into 12 sub-intervals.

• According to the HSV color quantization sub-interval, the dynamic detection bird's nest image is divided into 1296 sub-images, among which:

  • When V is in the first sub-interval and the second sub-interval, the image detection area is black and is excluded.

  • When S is in the 1st sub-interval and V is in the 10th sub-interval and 11th sub-interval, the image detection area is white and is excluded.

  • The area outside the other subintervals is a colored area, which is retained and the nest area is searched.

• By collecting and analyzing the power detection image whose tone is most similar to the bird's nest in the image detection area, the H, S, and V components of the bird's nest area are mainly concentrated at [100,120], [0,80], and [40,200] respectively.

Evaluation of Detection results

• Preprocess the bird's nest image detected by the drone. After detecting the bird's nest target, the detection effect of the bird's nest area can be evaluated. The data sources of the nested image detection result evaluation method include two metrics based on the single-shot detector convolutional neural network and the HSV color space.

• First, the measurement index of the Bird's Nest power inspection image based on the deep convolutional neural network based on IOU (Intersection Over Union); second, the measurement of the detection model is based on classification/localization loss. IOU (Intersection over Union) is a standard that measures the accuracy of detecting corresponding objects in specific data sets, such as the PASCAL VOC data set and COCO data set. This article is based on the format of the PASCAL VOC data set, uses the IOU standard to calculate the target detection results of nested images, and calculates the confidence of the prediction.

• Images are detected by SSD and then color space filtered. The filtered bird's nest image result is measured by PP, which represents the space occupied by the actual pixels of the bird's nest in the bird's nest image detection ROI, and represents the percentage of pixels belonging to the bird's nest. This article proposes to use pixel percentage PP to express the filtering results. The calculation formula of PP is as follows:

  • $$PP=\frac{\sum pcount}{width\ast height}$$

  • On top of this, pcount represents each pixel detected by the H, S and V components. Among them, width and height represent the image width and height of the ROI image respectively. Use PP to count the number of filter results, and then evaluate the detection results.

• There are four types of test results: TP, TN, FP, and FN. After calculating these metrics, the number of four types of results in the final nest detection is calculated. The bird's nest image detection method was comprehensively evaluated on precision, recall rate and f-value. The F value is a mixed value between Precision and Recall and is used to comprehensively evaluate the effect of classification or detection. Through the calculation of the above parameters, the final result of the bird's nest detection is obtained.

EXPERIMENT AND ANALYSIS

• This paper preprocesses the power inspection image data set and applies the Single Shot Detector target detection network to the bird's nest image detection of power inspection. Then, use OpenCV to build an image processing tool, construct an area filter in color space, and filter the area of ​​the nest.

Experimental environment and data

• The software and hardware configuration of the experimental computing platform are as follows: Win 10, Intel Core i9-7920X @ 2.90 ghz12cpu, GTX 1080 Ti gpu, CUDA 9.0, cuDNN 6.0, 32GB memory, Tensorflow-gpu version 1.9.0.

• The experimental dataset consists of 971 labeled images of power patrol bird's nests used for training and testing. According to a certain ratio, all images are divided into two parts: 650 training images and 321 testing images. and conducted experiments on the dataset.

Results and Analysis

• The bird's nest detection method proposed in this article mainly includes two parts: 1) detection of the target network; 2) filtering of the ROI image.

• The Shot Detector object detection model is trained to preload the weights trained in the COCO dataset, and then the network is fine-tuned using the local nest dataset. Further, set the batchsize to online learning, set the learning rate to 0.0003, decay the learning rate to 0.0001 when training to 20000, set the image scaling to 300×300, set non-maximum suppression (NMS) to 200, positive and negative samples The thresholds are set to 0.75 and 0.25, the parameters are optimized using stochastic gradient descent (SGD), the number of training steps is set to 50000, and the model is recorded in batches. The classification and regression loss curves of the model with the number of training steps are shown in the figure below.

  • 

  • Training loss for classification and localization over training steps

• It can be clearly seen that as the training step size increases, the losses of classification and regression gradually decrease, and the model converges after certain fluctuations. After the step size reaches 50000, it is reduced to 0.15. Therefore, the 50,000-step model is frozen as a nest detection model. The average accuracy test of the model is 96.47%, and the AP test is 60.90%. As shown in the figure below, the bird's nest can be found effectively.

  • 

  • Precision-Recall curve

• This paper tested a total of 321 bird's nest images, including 331 nesting targets. The number of nest targets is 229, and the thresholds of IOUs and Scores are 0.5 and 0.9 respectively. Among them, 223 nests were correctly identified, 104 nests were not identified, and 6 nests were incorrectly identified. Then, the method based on HSV color space is used to filter the ROI area of ​​the nest, and the PP value is set to 10%. Finally, 227 nested ROI regions were selected from the 229 detected nested ROI regions, of which 223 were correctly identified and 4 were incorrectly identified. The bird's nest detection accuracy is as high as 98.23%, and the recall rate is still 67.37%.

• The tested image is shown in the figure below. It can be seen that the method in this article has a very good detection effect on bird's nests. It can be clearly seen from the image (a) below that the color of the bird's nest is mixed with the color of the background, and the proportion of objects in the bird's nest image is very small. In the picture below (b), the borders of the bird's nest are blurred in the sunlight, making the color of the bird's nest white. In the picture © below, the target of the bird's nest is blocked by the power tower and lacks sunlight, making the bird's nest more difficult to detect. Therefore, an SSD-based method is proposed to effectively detect bird nests in power line patrols.

  • 

  • Bird's nest detection results are used to detect images

Influence of different methods

• In order to verify the effectiveness of the bird's nest detection method proposed in this article, different detection methods were compared on the bird's nest data set under the same situation.

• As can be seen from the figure below, the accuracy of the HSV color space dropped to 39.27%, and the recall value was 36.1, which shows that the single application of the HSV color space algorithm performs awkwardly in power line bird's nest detection. Therefore, the hsv color space algorithm gets 37.61% on the F1 value.

  • 

  • Comparison of different detection methods

• The same values ​​for a single lens detector are shown similarly in Figure 8 (not found in the paper). It can be seen from the value of the single-lens detector that the accuracy reaches 96.47%, which is greatly improved compared with the method of hsv color space, and the recall rate reaches 67.3%, which is smaller than the method of hsv color space. With the improvement, the F1 value reached 79.33%.

• The above indicators of the method proposed in this article are the highest when combining the two methods. It can be seen that with the assistance of the HSV color space algorithm, the single-shot detector method can greatly improve the detection accuracy of bird's nests without changing the recall value, which needs further research.

CONCLUSION

• The bird's nest detection method in UAV powered patrol images proposed in this article combines a deep convolutional neural network with a color space. It takes advantage of the advanced image features of the convolutional neural network and the learning capabilities of the color features of the color space to compensate for the convolutional neural network. The robustness further improves the nested detection results. The method proposed in this article has an accuracy of 98.23% and an F-value of 79.92% on the verification data set. Compared with the single convolutional neural network method, the effectiveness of the method is proved. It has certain reference value for power self-inspection.