Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 5, No. 2, 2023 59 Smoking Driving Behavior Detection Based on Deep Learning Jinfan Huang1, *, Rong Li2 1 School of Computer Science, Yangtze University, Jingzhou Hubei 434000, China 2 Health Science Center, Yangtze university, Jingzhou Hubei 434023, China * Corresponding author: Jinfan Huang (Email: huangjflr@qq.com) Abstract: Smoking in driving not only reduces the accuracy of driving operation, but also leads to insufficient oxygen and higher possibility of traffic accident. Therefore, considering the safety of life and avoiding accidents as far as possible, a detection model based on deep learning which can quickly detect smoking driving behavior is designed. In this model, convolutional neural network is used to process the input frames of the video stream captured by the camera. After the shape feature extraction, fuzzy feature processing, motion feature detection and color feature region comparison, the smoking driving behavior can be judged. Through the design of a series of computer vision detection modules, not only can reduce the calculation of the model, but also improve the efficiency of deduction, so as to meet the performance requirements of real-time monitoring. In order to quickly find out the driver smoking behavior and trigger warning, so as to avoid unnecessary traffic accidents and ensure life safety. Keywords: Deep Learning, Computer Vision, Convolutional Neural Network, Smoking Driving. 1. Introduction With the continuous leap of science and technology, smoking detection technology is also constantly improved, detection algorithm is also greatly improved. The most traditional smoking detection method is smoke sensor, or wear physical equipment for detection, but this detection method has two defects. First, when the driver opens the window, certain smoke concentration will be diluted, and the sensor precision is not enough to detect it. Second, the detection equipment needs to be worn by everyone, which is not advisable from the perspective of cost and convenience. Another detection method is to make behavioral judgment by matching body movements with smoking movements. This method supports vector machine and other machine learning methods to judge the matching degree between smoking movements and body movements, so the accuracy of this detection method is relatively low. In addition to the physical method, there are traditional graphics in the target detection method to detect smoking, this method mainly carries out three parts: first set the step size and size of the sliding window, and then set the window in each position of the image for sliding. The characteristic information of smoking or smoke is extracted by using the direction gradient histogram method. Finally, the algorithm is used to classify and test the set window. After verification and comparison, the score with the highest test score is taken as the final test result. Due to the complexity of its classification algorithm, it requires a large amount of computation for each classification, and it is easy to be interfered by external substances, so it is not accurate in action positioning and not public in practical application. In this paper, a lightweight smoking detection model is designed by referring to the high-performance smoke detection algorithm. The accuracy of the detection is improved by extracting the feature image vector from different angles at multiple levels, processing the fuzzy degree of graphics, color detection and action inspection. The convolution kernel parameters of the model are reduced, so as to reduce the calculation amount of the model, and finally speed up the calculation speed and efficiency of the model to meet the real-time requirements. 2. Basic Detection Process and Extraction of Smoking Characteristics 2.1. Smoke shape feature detection Smoke is an obvious feature to identify whether a driver smokes or not. It can be seen from the area of smoke diffusion in the air, the irregular shape of smoke, the continuity of smoke, ambiguity and other related properties, which better reflect the basic characteristics of smoke. Due to the interference of external factors, smoke molecules will change irregularly in the process of air diffusion, and the edge contour of smoke is an irregular fuzzy shape which changes irregularly in real time. When it is verified that two two-dimensional figures with the same area, the circumference of the figure with irregular edge contour is obviously larger than that of the figure with regular edge contour. The relationship between contour and area can be analyzed according to the feature of circumference comparison when the area is the same, as shown in the figure 1 below: Figure 1. Contour and area contrast diagram Furthermore, through the relationship between the area in the edge contour of the target region and the edge perimeter, the complexity of the contour of the target region and the irregularity of the smoke can be quantified, and the complexity of the edge contour of the smoke can be obtained: 60 Permeter a Area (1) 2.2. Detection of smoke diffusion characteristics Diffusion is one of the most obvious features of smoke. It is detected by video sequence that the contour area of smoke in the air edge goes from zero to irregular and constantly enlarges. Conventional objects do not have diffusivity in the process of movement, such as driving cars, running cheetahs, etc., and their common characteristics are small or even unchanged in area during movement. Therefore, a large number of disturbing objects can be eliminated through smoke diffusivity. The growth rate of the target area can be expressed as: 𝑆 (2) It is known that the growth rate of smoke area will not exceed a range. Select the diffusion to threshold values P1 and P2 to interpret whether the diffusion degree is between P1 and P2. If it is within the diffusion threshold range, it is the target region, otherwise it belongs to the interference region. 2.3. Detection of smoke color characteristics The general color of smoke is gray and white, but when the combustible material burning large flame or combustion is insufficient, it will make the smoke from the original gray and white to gray and black. In addition, the gray and white smoke will appear partly translucent color and other color interference factors, so the difficulty of detection is far greater than the gray and black. It is not ideal to use the traditional single color detection model for block processing. The mixed color model based on RGB, HSV and HIS is used to segment the smoke region, and the color saturation, color brightness and color region difference of the pixels in the smoke image are used to achieve the purpose of recognition. The first is the almost RGB color model. Since the smoke color is grey and white, there is little difference between the three values in the RGB space of the smoke color. The difference threshold between the maximum and minimum values in the mean range of the RGB component values can be used as the criterion. Therefore, in RGB color space, smoke color meets: |π‘€π‘Žπ‘₯ 𝑅, 𝐺, 𝐡 𝑀𝑖𝑛 𝑅, 𝐺, 𝐡 | 𝑀 (3) 𝑀 𝐴𝑣𝑔 𝑅, 𝐺, 𝐡 𝑀 (4) The second is based on the HSV color model. Due to the different shades of smoke color, the degree of diffuse reflection in the HSV features is different. The brightness V of the HSV space of smoke color gradually declines from inside to outside. Moreover, it is the influence of temperature. Most smoke colors are light blue when they are low, so H changes in the blue area. Therefore, smoke color can be judged according to the threshold range between brightness V and hue H in HSV. In the space of smoke color HSV, this formula should be satisfied: 𝐻 ∈ 𝐻 , 𝐻 , (5) 𝑆 ∈ 𝑆 , 𝑆 , (6) 𝑉 ∈ 𝑉 , 𝑉 (7) Third, based on HIS color model, it is known that smoke color has semi-transparency. Therefore, the color saturation is higher in the area with high smoke concentration, and the color saturation is lower in the area with low concentration at the edge of smoke. When the concentration of smoke changes from thick to thin, its saturation should be high to low. And the threshold value of smoke saturation changes between saturation S and saturation I, which is also the basis for judging the color of smoke. In HIS color space, smoke satisfies this formula: 𝑆 ∈ 𝑆 , 𝑆 , (8) 𝐼 ∈ 𝐼 , 𝐼 (9) Finally, the smoke regions divided by RGB, HSV and HIS are binarized respectively, and then the binarized graphics are operated logically to segment and detect the smoke in the video. 3. Smoke Detection algorithm 3.1. Video frame conversion According to the resolution, each frame in the smoke video image is preprocessed respectively from the model conversion of the smoke color, the filter color of the image, the gray degree of the image, the external illumination factor and the reduction of noise, so that the difference of the processing method can be determined by comparing the color of the smoke. This improves algorithm performance and reduces false positives. The algorithm also includes motion detection and frame conversion to binary, reducing unwanted brightness factors and preparing for subsequent smoke recognition. 3.2. Motion detection and color conversion Motion detection is carried out on the preprocessed smoke image, the steps of extracting the target of each frame are extracted, each frame detected is stored as the background, and the difference between the sequential frames is found. The use of subtraction between sequence frames using specific steps increases processing speed and reduces processing time, thus facilitating the detection of differences between frames and determining the final goal of each frame. The tracking mechanism for each object to determine the coordinates to store is labeled: 𝑑 , π‘Žπ‘π‘  𝐢 π‘₯, 𝑦, 𝑛 𝐢 π‘₯, 𝑦, 𝑛 (10) C represents image sequence number, n represents frame number, where d>t is taken as extraction object, when d>t, i = i+1. In the extraction algorithm, when each object tests the frame, the frame will be converted to HSV color space, so as to determine the value from all objects in the frame. The V value can be obtained from the following formula: V max r x, y , g i, j , b i, j (11) 61 3.3. Standard deviation value for smoke detection After the target is extracted, the smoke is individually examined, and it varies in brightness and color over a range of ranges, from a clear grayish white to a grayish blue. In the analysis of smoke intensity region, the smoke color condition value of RGB color space is determined by 𝑅 π‘Ž 𝐺 π‘Ž 𝐡 π‘Ž (12) Get, indicating that the three components (RGB) of the smoke pixel are equal. In order to determine the color attribute of the target region, the tracking technology is used to locate and store the coordinates of each object, and the standard deviation of each component (R,G,B) is used in formula (13) to measure the approximation between each value. Formula (14) can also be used to calculate the maximum (std) value of the color component (R,G,B) for threshold based testing so that colored smoke targets are identified. 𝑠𝑑𝑑 π‘₯ βˆ‘ βˆ‘ πœ‡ (13) Where, X represents the vector, i represents the RGB component, represents the mean value, and obtains the maximum standard deviation of the RGB component. 𝑆𝑑𝑑 max 𝑠𝑑𝑑 π‘Ÿ , 𝑠𝑑𝑑 𝑔 , 𝑠𝑑𝑑 𝑏 (14) Where, when the standard deviation is very small, the target represents the smoke property. The threshold is applied when the following conditions are met, and the target represents an area of interest that contains grayscale and non-grayscale gradients of the smoke. π‘“π‘’π‘Žπ‘Ÿπ‘‘π‘’π‘Ÿπ‘’1: 𝑖𝑓 𝑆𝑑𝑑 π‘‘β„Ž π‘‘β„Žπ‘’π‘› π‘‘β„Žπ‘’ π‘œπ‘π‘—π‘’π‘π‘‘ 𝑖𝑠 π‘ π‘–π‘”π‘›π‘–π‘“π‘–π‘π‘Žπ‘›π‘‘ π‘Ÿπ‘’π‘”π‘’π‘–π‘œπ‘› (15) 3.4. Smoke transparency was measured by Beaulieu transform Beaulieux algorithm is often used for data analysis, because it decomposes the signal into sinusoidal components of different frequencies, it is known as an effective algorithm. It is mainly used in the fields of image processing and signal processing, and its applications range from frequency analysis, filtering and convolution to power spectrum and estimation. As shown in Figure 1 and 2, it can be noticed that some colors in the smoke image have transparency. This algorithm uses formula (16) to convert the maximum value of each component of RGB to calculate transparency. In order to improve the accuracy of smoke transparency detection, Beauliffe transform is used to detect transparency according to formula (17) and (18) : 𝐼 max π‘Ÿ, 𝑔, 𝑏 (16) 𝐼 𝐹𝐹𝑇 𝐼 (17) 𝑝 𝐼 βˆ‘ 𝑛𝑗/π‘˜ (18) Where nj is the frequency, j = 1,2... k, where k is the frequency number, two object features can be used to identify object transparency, the first is the value of the maximum peak value of the histogram, and the second is the change value between figure 2. Figure 2. The offset value of the image 4. Experimental Results and Analysis 4.1. Experimental Results MATLAB is used to test the selected video. There are three important steps in video analysis, which are detection of smoke moving target, frame tracking and target trajectory analysis to identify target behavior. Calculated at 25 frames per second, the video is normalized to 320 pixels by 240 pixels, and the smoke object is displayed in a green border. video(1) video(2) video(3) video(4) video(5) video(6) video(7) video(8) Figure 3. Smoke detection chart The smoke detection status is shown in Table 1: Table 1. Smoke detection status Video serial number Smoke condition Description video(1) N Moving man video(2) Y Only smoke video(3) Y Only smoke video(4) Y Only smoke video(5) Y Only smoke video(6) N Moving car video(7) Y Only smoke video(8) Y Smoke man 62 4.2. Experimental analysis In this experiment, a smoke detection method based on frame motion was improved by early smoke analysis. The background and different modeling methods are used to detect the moving target of each frame, and the moving image is converted into binary mode to remove unnecessary brightness pixels. Two features of smoke detection, gray feature and transparency feature, were used to detect transparent smoke by Boli leaf, and the mean and standard deviation were calculated to detect gray smoke. The method in this paper was tested on multiple videos, and the accuracy reached more than 90%, as shown in Table 2 : Table 2. characteristics of video frame number conversion times Video serial number Framework Cigarette Smoke Video(1) 630 615 15 Video(2) 1853 1378 457 Video(3) 552 156 396 Video(4) 168 0 0 Video(5) 629 629 1 Video(6) 356 0 0 Video(7) 1168 1168 1 Video(8) 614 614 1 5. Conclusion According to the actual application scenario of smoking driving behavior, this paper proposes a smoking driving detection model based on depth science which can quickly detect smoke and give warning. The model has a good detection effect on the transparency and regional segmentation of smoke. On the one hand, solve the problem from the real-time model, and carry on the multi-dimensional analysis and optimization of the extracted target detection image features, which not only reduces the calculation amount of the model, but also speeds up the computational speed and accuracy of the recursive model. On the other hand, from the outdoor smoke detection, especially transparent smoke using Bolye algorithm, improve the detection ability and accuracy; At the same time, target tracking and edge method are used to improve the accuracy of target location. Finally, context information is used to reduce the false detection rate of target objects. In this paper, the model proposed by self-made training text verifies that the model has a good detection effect. Due to the diversity of data sets, it is not ideal in practical application. In the future, the training will be further enhanced to reduce errors and improve the detection effect of the model in actual industry. Acknowledgment We thank Rong Li for reviewing the article. References [1] LinYL Dollar P, Girshick RB, et al. Feature pyramid networks for object detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, July 21-26, 2017. Washington: IEEE Computer Society, 2017,1(2): 4.W.-K. Chen, Linear Networks and Systems (Book style). Belmont, CA: Wadsworth, 1993, pp. 123–135. [2] Woo S, Park JC, Lee JY, et al. CBAM: Convolutional block attention module[C]//LNCS 11211: Proceedings of the 15th European Conference on Computer Vision, Munich, September 8-14, 2018. Heidelberg: Springer, 2018, 3-19.B. Smith, β€œAn approach to graphs of linear forms (Unpublished work style),” unpublished. [3] Yun S, Han D, Chun S, et al. CutMix: Regularization strategy to train strong classifiers with localizable features.[C]// Proceedings of the IEEE International Conference on Computer Vision, Seoul, October 27- November 2, 2019. Washington: IEEE Computer Society, 2019, 6023–6032. [4] Zheng ZH, Wang P, Liu W, et al. Distance-IoU Loss: Faster and better learning for bounding box regression.[J] Proceedings of the 34th AAAI Conference on Artifificial Intelligence, 2020, 34(7): 12993-13000. [5] Senyurek VY , Imtiaz MH , Belsare P , et al. Smoking detection based on regularity analysis of hand to mouth gestures[J]. Biomedical Signal Processing and Control, 2019, 51: 106-112. [6] Senyurek V, Imtiaz M, et al. Cigarette Smoking Detection with An Inertial Sensor and A Smart Lighter[J]. Sensors, 2019, 19(3): 570-588. [7] Ye S, Bai Z, Chen H, et al. An effective algorithm to detect both smoke and flame using color and wavelet analysis[J]. Pattern Recognition and Image Analysis, 2017, 27: 131-138. [8] Yan B, Zhang JL. Research on Image Translation Based on Generative Adversarial Network[J]. Foreign Electronic Measurement Technology, 2019,38 (6)130-134. [9] Sun X, Li XG, Li JF, et al. Research Progress of image super resolution Restoration Based on Deep Learning [J]. Acta Automatica Sinica, 2017,43(5):697-709. [10] Du QG, Zhai XC, Wen Q, et al. Recursive Error Analysis of Complex Assembly Based on Rigid Body Kinematics[J]. Journal of South China University of Technology (Natural Science Edition), 2017,45 (9):26-33. [11] Zhao M, Zhang W, Wang X, et al. Smoke detection based on spatio-temporal background model combined with various texture features[J]. Journal of Xi 'an Jiaotong University, 2018, 52(8):72-78. [12] Muhammad K, Ahmad J, Mehmood I, et al. Convolutional neural networks based fire detection in surveillance videos[J]. Ieee Access, 2018, 6: 18174-18183.