Applied Science and Innovative Research ISSN 2474-4972 (Print) ISSN 2474-4980 (Online) Vol. 1, No. 2, 2017 www.scholink.org/ojs/index.php/asir 80 An Agricultural Spraying Robot Based on the Machine Vision Zhongrui Wang1 & Zhongcheng Wang2* 1 Qingdao No.2 Middle School of Shangdong Province, Qingdao, China 2 Qingdao Agricultural University, Qingdao, China * Zhongcheng Wang, E-mail: wangzc0826@163.com Received: May 20, 2017 Accepted: June 2, 2017 Online Published: June 6, 2017 doi:10.22158/asir.v1n2p80 URL: http://dx.doi.org/10.22158/asir.v1n2p80 Abstract Accurate target spraying is a key technology in modern and intelligent agriculture. For solving the problems of pesticide waste and poisoning in the spraying process, a spraying robot based on binocular machine vision was proposed in this paper. A digital signal processor was used to identify and locate tomatoes as well as to control the nozzle spray. A stereoscopic vision model was established, and color normalization, 2G-R-B, was adopted to implement background segmentation between plants and soil. As for the tomatoes and plants, depth information and circularity depended on the nozzle’s target, and the plant shape area determined the amount of pesticide. Experiments shown that the recognition rate of this spraying robot was up to 92.5% for tomatoes. Keywords tomatoes, spraying robot, machine vision 1. Introduction According to agricultural departments, there are approximately 35 varieties of and 58 million manual plant protection machines in China (Ling & Fenfen, 2002). Overseas, spray technologies are more advanced than domestic varieties, such as electrostatic aerial spraying (Latheef, Carlton, Kirk, & Hoffmann, 2009). However, all of these sprayers and technologies are widely used at present at home and abroad that do not emphasize precision or pilot status, waste a large amount of pesticides and are harmful to workers’ health. Thousands of intoxication accidents have occurred in the past (Xiongkui, 2004). Herbicide abuse also hurts the environment in addition to leading to soil acidification and water pollution (Peralta & Hegazy, 1994). Traditional backpack sprayers, which have a high failure rate and poor performance, could not meet the requirements for scientific and reasonable spraying (Giles & Slaughter, 1997). Some studies on auto-spraying have been performed by domestic and international scholars (Yiming, 2004). However, at present, most spraying robots lean towards weed identification instead of crop identification, such as tomatoes. In addition, most spraying robots based on machine vision are www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 81 Published by SCHOLINK INC. developed using OpenCV, which depends on VC6.0 to be run an industrial computer or PC, leading to a series of problems due to the long system development cycle, high development cost, large volume and high power consumption. More importantly, existing systems use a single camera to identify plants; however, the obtained images are not good enough to generate accurate depth information, and thus, the precision of locating and spraying targets is extremely low. To overcome the disadvantages listed above, a spraying robot based on location using binocular machine vision is presented in this paper. The digital signal processor TMS320DM642 is used to identify and locate tomatoes rather than OpenCV, an open source computer vision library based on a PC. Referencing the binocular vision model, we designed and implanted a location algorithm over the DSP. In addition, we proposed and verified the tomato features extraction algorithm. 2. Hardware Design The spraying robot consists of two parts: a host computer image process module and a slave computer control module. The image process module includes the image process core, TMS320DM642; a power module; a binocular vision camera; and an image output module. The slave computer control module includes the main control module, TMS320F2812; a power module; a manipulator; a spraying robot; a GSM module; a GPS module; and a mobile platform. The sprayer works as follows. First, the binocular vision camera captures images from the head area and transmits them to the host computer; the TMS320DM642 image processor (DM642 in short) uses color normalization 2G-R-B to implement background segmentation between plants and soil while it uses circularity to identify tomatoes from other plants. Additionally, the binocular locating model could acquire the targets’ depth and three-dimensional coordinate information. Then, the three-dimensional coordinate data are transported to the slave computer through the RS232 serial communication. TMS320F2812 (F2812 in short) is the main control chip; aims the nozzle, which is fixed on the terminal of the manipulator, at the targets; and then starts the spray system to accurately spray the targets, depending on the received data from the host computer. Furthermore, TMS320F2812 could control the amount of pesticide using the area of the tomatoes. In addition, a GSM module was used to transmit data regarding the machine conditions to the server, and the user could access server by a mobile phone. In addition, a GPS module is used as a high accuracy positioning module; hence, workers can easily adjust the path of the machine. Therefore, the spraying machine is able to achieve unmanned and intelligent spraying. 3. Software Design 3.1 Main Control Module Program Design The main control module mainly performs 3 functions. 1) Receiving and analyzing the image process results from TMS320DM642 as well as controlling the spraying robot to spray quantitatively toward the targets. www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 82 Published by SCHOLINK INC. 2) Driving the mobile platform to move and stop precisely. 3) Gathering the system states and position information, receiving the request from the worker and transmitting information to server. The main control chip F2812 that drives the mobile platform, receives and analyzes the image process results from DM642, controls the manipulator and sprays the targets, responds to the requests from the worker and sends system state and poison information. 3.2 Main Image Processing Module Program The diagram of the main image processing module is shown in Figure 1. Using a binocular camera to capture images, we first process the images through the color normalization 2G-R-B and the Laplacian algorithms. Furthermore, by referencing the binocular locator model, the DSP could calculate the three-dimensional coordinate values after extracting the targets’ features. image capture image Pre - processing shape feature extraction display and output binocular vision locating end start Laplace edge processing Figure 1. Main Image Processing Module 3.3 Section and Subsection Headings 3.3.1 Color Normalization 2G-R-B A MJW camera by SONY Corporation was adopted to capture two-channel images. The captured images include R, G and B three color components, which, respectively, represent red, green and blue. According to previous research by Woebbecke and Meyer (Meyer et al., 1988), the color parameter G makes a great difference between green plants and soil. In addition, excess green 2G-R-B, similar to R-G-B, which aggravates the green parameter, is an effective color parameter for background segmentation of plant images (Woebbeckew, Meyer, Mortensen, & Von Barge, 1995).                             others,,,5.2 255,,,5.2255 ,,,20 , ， ， ， yxByxRyxG yxByxRyxG yxByxRyxG yxExG (1) www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 83 Published by SCHOLINK INC. Parameter  yxExG , represents the useful image, and three parameters,  yxG , ,  yxR , and  yxB , represent three-color components matrixes (Ying, 2005). 3.3.2 Laplace Edge Processing The Laplacian equation uses second derivative information and has strait isotropic homogeneity. The image becomes a cliffy zero cross point, which allows for judgment of the target’s edge to be made easily. The definition of the Laplacian equation is:   2 2 2 2 2 , y f x f yxf       (2) The images processed by the Laplacian equation are shown in Figure 2. It is obvious that the eight neighborhoods Laplacian processed image has more practical values than the four neighborhoods Laplacian image. (1) The original image (2) Eight neighborhoods image Figure 2. Laplacian Edge Processing 3.3.3 Tomatoes’ Features Extraction The tomatoes have an approximate ball shape, which is different from the shape of the plants, and the area of a tomato is obviously less than that of a plant. Therefore, we use the area and circularity parameters to distinguish tomatoes from plants. Circularity is a characteristic quantity to calculate the complex degree of the shape based on the area and perimeter. The circularity calculation formula is: 2 4 L S e    (3) Where parameter S represents the region area and L represents the region perimeter. The smaller the circularity, the greater the complexity of the image, and vice versa. When parameter e is close to 1, it is implied that the image is extremely close to a circle. The steps of identify algorithm for tomatoes are as follows: Ⅰ. In the first image captured by the left stereo camera, search for the first point of the targets’ contour, www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 84 Published by SCHOLINK INC. process it using the Laplacian equation above and mark it as coordinate 1A . Ⅱ. Skipping 1M points in the horizontal direction, whose magnitude is decided by the targets’ size, search for the second point in the contour and mark it as 1B ; if there is not a second point, then repeat step Ⅰ. Ⅲ. Skipping 1N points in the vertical direction, start from point 1A and search for the third point in the contour and mark it as 1C ; if the query process is hindered, then repeat step Ⅱ a maximum of 10 times until point 1C is determined. If a second point does not exist, then return and repeat step Ⅰ. Similar to point 1M , the magnitude of parameter 1N is decided by the target’s size. Ⅳ. Make a vertical bisector function through the 1A , 1B and 1C that were found and mark it as function 1f ; Similar to 1A and 1C , the function is denoted by 2f . Ⅴ. Seek the coordinates of the intersection point between function 1f and 2f , mark it 1O , which is the target’s centroid. Ⅵ. Correspondingly, process the second image captured by the right stereo camera and calculate the target’s three points, which are 2A , 2B , 2C , and the centroid 2O . The schematic diagram of the algorithm is shown in Figure 3. 1f 2f A B C O M N Figure 3. Schematic Diagram of the Algorithm 3.3.4 Tomatoes’ Volume Calculation In the steps Ⅰ and Ⅴ of section 3.3.3, we can obtain the length of 1A and 1O , which is the target’s radius and is denoted by 1r . Similarly, compute 2r through 2A and 2O and average r through 1r and 2r . We could also use the formula 3 3 4 RV  to calculate the target’s volume, which is denoted by the parameter V , and decide the amount of pesticide for the target. www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 85 Published by SCHOLINK INC. 3.4 Binocular Locating Arithmetic Design 3.4.1 Camera Calibration Camera calibration is the process by which the intrinsic parameters and external parameters of the binocular camera are captured. In this design, the lattice calibration board is 7×5. Using a binocular vision camera, 20 images were captured from a calibration board that was placed at different locations. We use the sixth calibrated parameters from the calibration board as the external parameters. 3.4.2 Stereo Matching Based on Feature Points In section 3.3.3, we calculated the feature points of the centroid   111 , yxO from the left image and the centroid  222 , yxO from the right image. These feature points likely represent the same target if coordinate 1y is close to coordinate 2y . Therefore, the stereo matching courses become extremely simple and efficient. 3.4.3 Three-Dimensional Coordinate Calculation Figure 4 shows the binocular stereo vision model of arbitrary point P in space. LO and RO are the left and right camera coordinate system origins, respectively, while 1I and 2I are the image planes of the left and right camera, respectively. P OL ZR I1 I2P1 P2 XR OR XL ZL YL YR Figure 4. Binocular Stereo Vision Model The centroids are 1O and 2O , whose images coordinates, respectively, are  111 ,vup and  222 ,vup , which were calculated in section 3.3.3. According to the camera imaging model, we could obtain the formulas as follows (Zou, X. J., Zou, H. X., & Lu, 2012).                                               11 1 34333231 24232221 14131211 x y x mmmm mmmm mmmm z y x Mv u z llll llll llll ll l l (4) www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 86 Published by SCHOLINK INC.                                               11 1 34333231 24232221 14131211 x y x mmmm mmmm mmmm z y x Mv u z rrrr rrrr rrrr rr r r (5) lM and rM are the projection matrices, and  zyx ,, is the three-dimensional coordinate of point P . Eliminating lZ and rZ from the above formulas, we can obtain the arithmetic expression below.                        34333231 24232221 34333231 14131211 34333231 24232221 34333231 14131211 rrrr rrrr r rrrr rrrr r llll llll l llll llll l mzmymxm mzmymxm v mzmymxm mzmymxm u mzmymxm mzmymxm v mzmymxm mzmymxm u (6) The formula above is changed into matrix form: bAP  (7)                  233322322131 133312321131 233322322131 133312321131 rrrrrrrrr rrrrrrrrr lllllllll lllllllll mvmmvmmvm mummummum mvmmvmmvm mummummum A (8)  TzyxP  (9)                  rrr rrr lll lll vmm umm vmm umm b 3424 3414 3424 3414 (10) According to least squares method, we can determine the concrete three-dimensional value.   bAAAP T1 '   (11) 4. Test Result and Analysis 4.1 Test Site and Condition Selection The prototype of the spraying robot is shown in Figure 5. It was placed in the middle of two trenches of plants; the binocular camera is between 30 cm and 80 cm away from the targets. The diameters of the tomatoes in the greenhouse range from 6 cm to 12 cm, which is beneficial for testing and verifying the generality of our feature extraction algorithm. To test the adaptability of the system, the experiment was conducted in a tomato greenhouse from the morning, with enough light, to dusk, with less light, with no illumination compensation, and the experiment was repeated several times. www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 87 Published by SCHOLINK INC. Figure 5. The Spraying Robot Prototype 4.2 Test Results and Analysis The binocular vision camera captures images from the head area and transmits them to the host computer; DM642 processes and analyzes the images and calculates the target’s three-dimensional value depending on the equations in caption 3.4.3. Then, the three-dimensional coordinate data can be transported to F2812. We use a six Degrees-Of-Freedom (DOF) manipulator for target spraying and use the relay to control the pesticide spraying robot. In detail, F2812 decides when to move or not move the mobile platform; the manipulator decides which target to spray, and the relay determines when the pesticide system sprays and stop spraying. To test the illumination factor that influenced this system, we settled the system in a tomato greenhouse from 8:00 a.m. to 8:00 p.m. and tested it every two hours. Even at dusk, we did not add any illumination compensation. Figure 6 demonstrates that in the morning and especially at dusk and night, the identification rate was lower than at noon. The identification rate is stable both after and before noon, such as from 10:00 to 16:00. Obviously, illumination is the most vital factor for the identification rate because the camera is able to capture less image information when with less illumination. 0 20 40 60 80 100 8: 00 10 :0 0 12 :0 0 14 :0 0 16 :0 0 18 :0 0 20 :0 0 T ime/h Id e n ti fi c a ti o n r a te /% 1d 2d 3d 4d 5d Figure 6. The Influence of the Illumination Intensity www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 88 Published by SCHOLINK INC. Figure 7 shows that when the targets’ diameter is less than 7 cm and more than 12 cm, the identification rate declines repeatedly. However, in a relatively large area between 7 cm and 12 cm, the success rate is considerable and reliable. Using Figure 10, we can conclude that the targets’ diameter is an important factor for the identification rate. Therefore, the feature extraction algorithm would lose efficacy when the target’s diameter is too small or too large to be captured by the camera. 0 20 40 60 80 100 6 7 8 9 10 11 12 13 Target diameter/cm Id e n ti fi c a ti o n r a te /% 1d 2d 2d 4d 5d Figure 7. The Influence of the Target Diameter 5. Conclusions In this paper, DM642 was used as the main control core to identify and locate tomatoes, while F2812 was used to control the nozzle spray. Tests demonstrated that the illumination intensity, target diameter and target distance were the three factors that affected and decided the identification rate of the system. The system spray design in this paper is able to effectively reduce worker’s intoxication accidents, spray cost, and pesticide residue, which is of great significance for the development of agricultural spraying towards intelligent systems, modernization, and precision. On the one hand, what appropriate conditions and circumstances should be developed as much as possible. On the other hand, for these problems, illumination compensation and a more general identification algorithms are necessary and urgent. References Giles, D. K., & Slaughter, D. C. (1997). Precision Band Spraying with Machine-Vision Guidance and Adjustable Yaw Nozzles. Transactions of the ASAE, 40(1), 29-36. https://doi.org/10.13031/2013.21240 Latheef, M. A., Carlton, J. B., Kirk, I. W., & Hoffmann, W. C. (2009). Aerial electrostatic-charged sprays for deposition and efficacy against sweet potato whitefly (Bemisia tabaci) on cotton. Pest Management Science, 65(7), 744-752. https://doi.org/10.1002/ps.1748 Ling, Z., & Fenfen, D. (2002). Actuality and Development Trend about Equipment for Crop Protection and the Spray Technique of Agricultural Chemicals in China. Chinese Agricultural Mechanization, 6, 34-35. www.scholink.org/ojs/index.php/asir Applied Science and Innovative Research Vol. 1, No. 2, 2017 89 Published by SCHOLINK INC. Meyer, G. E. et al. (1988). Electronic image analysis of crop residue cover on soil. Transaction of the ASAE, 31(3), 968-973. https://doi.org/10.13031/2013.30808 Peralta, R. C., & Hegazy, M. A. (1994). Musharrafieh, GR. Prevention Pesticide Contamination of Groundwater While Maximizing Irrigated Crop Yield. Water Resources Research, 30(11), 3183-3193. https://doi.org/10.1029/94WR01724 Woebbeckew, D. M., Meyer, G. E., Mortensen, D, A., & Von Bargen, K. (1995). Shape features for identifying young weeds using image analysis. Transaction of the ASAE, 38(1), 271-281. https://doi.org/10.13031/2013.27839 Xiongkui, H. (2004). Improving Severe Draggling Actuality Of Plant Protection Machinery and Its Application Techniques. Transactions of the Chinese Society of Agricultural Engineering, 20(1), 13-15. Yiming, W. (2004). Study on Infield Weed Detection Using Machine Vision. China Agricultural University. Ying, P. (2005). Research on identification of nightshade crops and weeds based on machine vision. Jangsu University. Zou, X. J., Zou, H. X., & Lu, J. (2012). Virtual Manipulator-based Binocular Stereo Vision Positioning System and Errors Modelling. Machine Vision and Applications, 23(1), 43-63. https://doi.org/10.1007/s00138-010-0291-y