GPU-enabled pavement distress image classification in real time

22 Pavement assessment is a crucial process for the maintenance of municipal roads. However, the 23 detection of pavement distress is usually performed either manually or offline, which is not only 24 time-consuming and subjective, but also results in an enormous amount of data being stored 25 persistently before processing. State-of-the-art pavement image processing methods executed on 26 a CPU are not able to analyze pavement images in real time. To compensate this limitation of the 27 methods, we propose an automated approach for pavement distress detection. In particular, GPU 28 implementations of a noise removal, a background correction and a pavement distress detection 29 method were developed. The median filter and the top-hat transform are used to remove noise 30 and shadows in the images. The wavelet transform is applied in order to calculate a descriptor 31 value for classification purposes. The approach was tested on 1549 images. The results show that 32 real-time pre-processing and analysis are possible.


INTRODUCTION
In recent years, the condition of municipal roads has deteriorated rapidly, leading to increased 36 fuel consumption, thus increased emissions and environmental pollution, and even greater 37 number of vehicle damages and traffic accidents [Spielman 2014]. To reduce the negative impact 38 of deteriorated roads on the driving quality, roads need to be maintained in good condition, for 39 example by repairing parts of the road surface where pavement distress, visible as cracks or 40 potholes, is present. For this purpose, knowledge about the exact location of pavement distress is 41 required and pavement assessment is an essential task [Orr 2015].  Such methods could be employed in order to store only those images on which distress had been 65 identified and discard all other images without distress, resulting in less memory requirements 66 and less subsequent processing time needed compared to the state-of-the-art case. 67 However, although the central processing unit (CPU) technology has evolved during the last 68 decade, modern CPUs are still not able to cope with the requirement of real-time execution of 69 related analysis methods, mainly due to the fact that image pre-processing is also needed. For 70 instance, noise removal as well as correction of non-uniform background illumination needs to 71 be applied to the images to enhance their quality in order to produce more accurate analysis 72 results. 73 Yet, the real-time processing requirement can be fulfilled by utilizing Graphics Processing Units 74 (GPUs). Applied not only for graphic operations, but also for computational tasks, GPUs have 75 proven their efficiency in diverse scientific fields in recent years [Owens et al. 2005]. 76 In this work, GPUs were used to accelerate the pre-processing and the analysis of pavement 77 surface images for the purpose of real-time pavement defect detection. In particular, a noise 78 removal method, a shadow removal method and an approach towards pavement analysis based 79 on the wavelet transform were implemented and validated. The next two sections provide information on state of practice and research concerning pavement 81 distress detection. Afterwards, GPUs are introduced. The approach is presented in thereafter, and 82 then the implementation is described. Performance tests were carried out to evaluate the 83 capability of the proposed implementation to process the images in real time. A case study was 84 performed to validate the approach and is described in the "Case Study" section. The paper 85 concludes with a summary of the main contributions and an outlook on future developments. kernel. Yet, the median filter is characterized by a high computational cost. The computational 132 complexity for sorting n values, a basic step within median filtering, with efficient sorting 133 algorithms is O(n*log n). 134 Another problem related to pavement images is the non-uniform background illumination. for the analysis only images taken under good weather conditions (i.e., when the weather was 142 overcast or mostly cloudy). However, the selection of the images is also a manual and time-143 consuming process and all images have to be stored before the analysis can begin. Zou et al.

144
[2012] presented a geodesic shadow-removal algorithm which is able to preserve the cracks in     As in the case with the median filter, the main drawback of the top-hat transform is its 169 computational complexity. The size of the structuring element required to preserve the edges or 170 details in the images leads to a vast number of pixels being considered for each anchor point.

172
A range of methods for distress detection in pavement images has been proposed in recent years.  ( 2 ) where (p,q) is the position of the coefficient in the corresponding subbands.

280
Then, the modulus is binarized according to Equation ( 3 ): where D is the binarized modulus and Cth is a threshold value estimated by wavelet thresholding.

283
Finally, HAWCP is calculated as where W and H represent the width and height of the image, respectively.

285
The HAWCP value ranges between 0 and 1 (or 0% and 100 %), where a value near 0 indicates a 286 good pavement surface, and high HAWCP values represent pavement distress.

302
In OpenCL, a single host is defined that is responsible for the coordination of code execution on 303 one or more devices [Gaster et al. 2013]. The host also interacts with the environment external to   which we try to address in this paper. First, pavement assessment is usually carried out either 325 manually or by using special dedicated vehicles. Second, the data acquired for pavement distress 326 detection is mostly processed offline, which results in a huge amount of data being stored 327 persistently.

328
To address the aforementioned problems, the following two research questions have to be This paper addresses the issues described previously by presenting an approach which is founded 336 on common vehicles. Instead of using dedicated vehicles, the idea pursued hereby is to use 337 vehicles which drive daily on the roads, such as buses and taxis. Nowadays, such vehicles are 338 equipped with built-in cameras, for example backup cameras, which can be used not only to 339 support the driver while parking, but also for other tasks, particularly in this case for road distress 340 detection.

341
In order to address the second research question, we propose online processing of pavement 342 images in real-time. With the aim of reducing storage consumption, only images which contain 343 distress will be stored, while images of good pavement surface will be discarded directly after 344 they have been taken and processed. However, to enable real-time pavement distress detection 345 while driving, either methods which do not require a long execution time need to be developed 346 or existing methods should be enhanced or implemented for faster architectures. In this work, 347 GPUs are utilized to enhance the performance of existing pavement image pre-processing and 348 analysis methods. As a result, real-time pavement distress detection is possible.

349
The approach proposed here is presented in Figure 4. To remove the noise, the images are first 350 convolved with a median filter. Second, the top-hat transform is applied to produce a more An overview of the implementation is depicted in Figure 6. First, the input image data that is    queues, the kernels, and set the kernel arguments, is also not considered, because these 448 initialization steps are executed only once at application startup and are not repeated for each 449 frame or image that has to be processed.

450
The following hardware was used for the performance evaluation tests: a 2.10 GHz Intel Core i7- in Figure 9. The transfer to the discrete GPU is significantly slower than the transfer to the 462 integrated GPU for large images.

463
The difference between the times required to transfer the HAWCP value of a single image is not 464 so considerable, because only one value needs to be transferred.

466
In our work, we used a median filter with a square structuring element of a size 3x3. The 467 execution times in milliseconds are shown in Table 1.

468
Top-hat transform 469 The top-hat transform was tested with a structuring element of a size 10x10. The performance 470 evaluation results are presented in Table 2  descriptor, is presented in Figure 10. implementations.

488
The total execution times for all image sizes are shown in Table 3. The speed-up calculated 489 according to Equation 10 is also presented. In case of the Nvidia GPU, the total execution time is 490 below 1.5 milliseconds. Theoretically, this allows processing more than 650 images per second.  potholes and patches were considered (Figure 13).

515
The images were manually labeled and ten-fold cross validation was performed in order to get a 516 reliable error estimate. For this purpose, the data was split into ten approximately equal 517 partitions. Each of these partitions was used for testing once, while the remaining data was used 518 for training. Three algorithms were used for classification, namely the C4.5 [Quinlan 1993 Perceptron, and 0.14 seconds for Rotation Forest.

25
The 5% of the images that were classified incorrectly are 77 images in total. Out of them, 15 525 images without distress were classified as images containing distress (false positives). In Figure   526 14, an example of a correctly classified intact pavement image (left) and an intact pavement 527 image that was incorrectly classified as image containing distress (right) is presented.

528
Nevertheless, this is still a promising classification result, because the objective of the rough 529 distress detection stage described in this paper is to identify potential distress locations. In a 530 further step, these potential locations will be assessed in detail by more comprehensive 531 algorithms.

532
Vice versa, the other 62 images which actually contain distress were classified as distress free 533 images (false negatives), mainly because of the different types of road surfaces considered in the 534 case study. Consequently, the locations these images were acquired at would not be taken into 535 account within the fine analysis. In order to counteract such errors, the methodology presented 536 here will be extended by incorporating textural features.  Yet, some images containing small cracks were incorrectly classified as good pavement images.

561
The approach presented in this paper can be improved by combining multiple descriptors to 562 obtain a more accurate representation of the distress. Future steps include the implementation of 563 other pavement distress detection techniques on the GPU, as well as the employment of Graphics

564
Processing Units for further pre-processing steps, such as the Bayer pattern de-mosaicing.