Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 76-82 76 https:doi.org/10.31763/businta.v4i2.283 Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season Aripriharta a,1,*, Adim Firmansah b,2, Nandang Mufti c,3, Gwo-Jiun Horng d,4 , Norzanah Rosmin e,5 a Department of Electrical Engineering, Faculty of Engineering, Universitas Negeri Malang, Indonesia b Graduate School, Faculty of Engineering, Universitas Negeri Malang, Indonesia c Department of Physics, Faculty of Mathematics and Natural Sciences, Universitas Negeri Malang, Indonesia d Department of Computer Science and Information Engineering Southern Taiwan University of Science and Technology, Taiwan e Centre of Electrical Energy System (CEES), Sekolah Kejuruteraan Elektrik, Fakulti Kejuruteraan, Universiti Teknologi Malaysia, Malaysia 1 aripriharta.ft@um.ac.id *, 2 adimfirmansah@gmail.com; 3 nandang.mufti.fmipa@um.ac.id , 4 grojium@gmail.com , 5 norzanah@utm.my * corresponding author 1. Introduction Palm oil is a plant species that grows in tropical areas. Palm oil is often used as a basic ingredient of products traded internationally, such as alternative energy sources [1] and vegetable oil [2]. In 2013, Indonesia's palm oil land reached 10.6 million ha. The number expands annually. In 2020, palm oil land has increased by up to 13.7 million ha [3]. The increase in demand and consumption of palm oil- based products leads the agriculture industry to explore the technology to provide benefits, competitiveness, and market viability [4]. Therefore, an appropriate management system is necessary, starting from land selection, nursery, planting, maintenance, and harvesting. The harvest system plays an essential role because it affects the freshness of the fruit produced. 1.1. Harvest system automation needs Harvest estimation is an important parameter in the agriculture industry to prepare the harvesting process, especially ordering transportation facilities. Companies need to calculate yields accurately so that there is data alignment of sales, purchases, and storage of harvest data. Companies generally use manual calculation methods to estimate harvest yields with the help of manual labors. In general, large companies cost around $600,000 to $1,000,000 for manual calculations per year [5]. In addition to manual calculation costs, many other evaluation costs such as overhead costs that include insurance and total equipment repair costs. Thus, companies need more sophisticated technology to calculate harvest yields automatically. A R T I C L E I N F O A B S T R A C T Article history Received August 1, 2020 Revised August 15, 2020 Accepted August 30, 2020 Harvest estimation is an essential parameter in the agriculture industries to estimate transportation facilities and storage areas in the harvesting season. Meanwhile, companies are required to calculate crop yields quickly and accurately. This paper reports on an experimental study in the form of a smart application to count oil palm fruit in the field quickly and accurately. The system used a single shot detector algorithm to count the number of fresh fruit bunches (FFB) on-site using a smartphone camera. The cutting area (CA) at the top of the collection was collected in various positions in the database. Our research documented that the algorithm matched the CA with the picture taken by the operator. Hence, the application automatically calculated the number of harvests per-site in the FFB unit. The data were then sent to the cloud database via a wireless router in a warehouse or through a cellular network. The main advantage of this application is reducing the theft that usually occurs on the spot. The model used performs very well for agricultural applications, with 94% to 99% accuracy. This is an open access article under the CC–BY-SA license. Keywords Image Matching Harvesting CNN Yolo mailto:nandang.mufti.fmipa@um.ac.id http://creativecommons.org/licenses/by-sa/4.0/ http://creativecommons.org/licenses/by-sa/4.0/ ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 77 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) Automatic crop yields counting provides many benefits, such as making the harvest process faster. A faster harvest cycle makes labor costs more efficient, especially laborers paid on an hourly basis. The automatic calculation system also provides more precise and consistent results. The low error rate can reduce the budget due to crop yields' miscalculations. The fast harvesting process also makes the company can produce fresh fruit. The automation system can reduce embezzlement of palm oil crops that harm the company. 1.2. Related Works Research on increasing crop harvesting efficiency has received much attention in the last few decades, especially in detecting and counting the number of fruits. One appliance widely used is computer vision techniques based on color Previous research carried out by [6] deployed image colors to detect maturity and count the number of green, oranges. The results showed a successful detection rate of 75.3%, with an error rate of 27.3% during the day. Sunlight conditions during the day have a significant influence on the results of detection [7], [8]. The sun's rays create shadows that cause a standard segmentation procedure to divide the surface into several fragments. Anchored by these results, [9] applied computer vision techniques at night to detect fruit using an artificial light source with 78% success rate. Meanwhile, Authors [7] used computer vision techniques for detecting oranges in outdoor environments-based color. The obtained accuracy rate is 76.5%. Furthermore, Authors [10] added size parameters to improve the accuracy of fruit detection. The accuracy level produced in their study using the color, shape, and size parameters increased to 90%. Traditional computer vision techniques have been widely adopted in the agricultural industry. Although the method works well under certain conditions, some cases require more precise detection due to changes in environmental conditions such as illuminations and occlusions [11]. The traditional computer vision algorithm also has a high error rate when detecting fruit that piled up. In recent years, the computing capabilities of electronic devices continue to increase. The processor continues to grow until it is created with a special purpose processor for graphics processing called the Graphics Processing Unit (GPU). In this era, Deep Learning (DL), which a sub-field of Machine Learning (ML), has been widely developed. These are included in the category of Artificial Intelligent (AI). DL uses a layered algorithm structure called an artificial neural network (ANN) [8]. DL makes it possible to build accurate detectors through training that automatically learns features from photos and uses them efficiently. In DL, visual features are generally extracted by the Convolutional Neural Network (CNN) [12]. In an earlier study, [13] adopted the Faster R-CNN model called Deep Fruit to detect fruit. The model produced after training has an accuracy of 83.8% for detecting paprika. With a different model, Authors [14] developed CNN based on Inception-ResNet architecture model called Deep Count to count fruit. This research was carried out by adding synthetic images to produce a better model. After tested in a real environment, the model has an accuracy of 80% to 85%. [8] employed a Single-Shot detector to detect fruit in real-time. The model also used synthetic images as training data and produced an accuracy of 0.9 with 0.64 Intersection over Union (IoU). All these three studies used a computer or minicomputer to conduct detection. Unlike the above three investigations, [15] applied the R-CNN Faster model and Single Shot Detector (SSD) to calculate crop yield estimates. In their study, the trained R-CNN Faster model was able to obtain an accuracy rate of 89%, while the SSD model had an accuracy rate of 82%. On the other hand, [16] applied DL in unmanned aerial vehicles (UAV) to count apples and oranges. They used two-layer CNN. The first layer is a fully convolutional network-based blob detector, and the second layer is a counting algorithm based convolutional network-based. Interestingly, this study not only implements DL but also combines linear regression to determine the results of the fruit calculation. The results showed the applied model had an accuracy rate of 91.3% for detecting apples. Based on the needs of the crop automation system and technological developments explained earlier in this paper, we aim to propose an automatic crop yield calculation as a new technological appliance in agricultural sectors. The proposed system applies the DL to calculate yields. In contrast to the study of the application of DL to calculate crop yields, our experiment uses a single shot detector model called You Only Look Once (YOLO) lite version 3 model. The model is embedded in the smartphone application to detect objects. Overall, system details are explained in the method section. 78 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) 2. Method This proposed system implemented a single-shot detector (YOLO) lite version 3 as CNN in a smartphone application to detect and count palm oil fruit. The smartphone application accessed the camera to take pictures. The system detected palm oil fruit in real-time and directly displays the bounding box on the detected fruit. Besides displayed on the smartphone application in real-time, the number of fruits is also sent to the cloud database. This system can use two network connection modes, namely a cellular network that utilizes a Base Transceiver Station (BTS) and a Wi-Fi network that utilizes a wireless router. Data sent in the cloud database is displayed in the form of a web interface that can be accessed via a computer or smartphone. The overall system block diagram is shown in Fig. 1. Fig. 1. System block diagram. This system can be used to calculate harvest yields that are outside the warehouse if there is a cellular network to send data to the database. Fig. 2 shows the proposed system implementation scenario. Workers who oversee counting fruit carry a smartphone with an installed automatic counter application. Workers who are in the warehouse area can also use Wi-Fi networks to send data to the cloud database. Besides, workers who work in outdoor areas can use the cellular network that is available to send data to the database. Fig. 2. Implementation scenario 3. Results and Discussion The test was carried out on a smartphone with a Mediatek MT6753T chipset. The smartphone uses an Octa-core 1.5 GHz Cortex-A53 processor and a Mali-T720MP3 GPU. Fig. 3 showed a screenshot of the application display when workers were counting the number of palm oil fruit. The detected fruit was marked with a red bounding box. The test results show that the used model can detect palm oil fruit well. ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 79 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) Fig. 3. Detection application Modifications to the smartphone application scripts were performed to display the confidence level of each detected fruit. Fig. 4 shows the display of the smartphone application that the script added to display the name and level of confidence of the detected palm oil fruit. The experimental results show a confidence level (accuracy) of 94% to 99%. The high level of confidence indicates the used model has great accuracy in detecting palm oil fruit. Fig. 4. Confidence value The detection button is used to start and end the palm oil fruit detection process. After the worker presses the detection button, the calculated number of palm oil fruit is automatically sent to the cloud database. The implementation of the proposed system can accelerate the process of harvesting palm oil. Manual calculations generally require ± 2 minutes. The time needed for manual calculations is also directly proportional to the number of fruits. Manual calculations for large numbers also have a high error rate. After implementing the proposed system, the calculation process of palm oil fruit decreased to ± 30 seconds, with a high degree of accuracy. Overall, the proposed system has an accuracy rate of 94% to 99%. A comparison of the proposed system accuracy with the previous research is shown in Table 1. The comparison results show that three studies can produce an accuracy rate above 90%. If the level of accuracy is greater, the error rate for fruit harvest calculation is thus lower. The level of accuracy is significantly influenced by the network architectures, datasets, training system, and the detected object form [8], [17]. The level of accuracy can be improved through the longer training stages of the model. The error level due to illumination can be minimized by providing a dataset with several illumination variations or add a process layer [18]–[20]. However, if more layers are used, then it takes longer computational time to detect fruit. 80 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) Table.1 Accuracy comparison Reference Model Accuracy (%) [13] Faster R-CNN 83,8 [14] Inception-ResNet 80-85 [8] Modified Yolo v2 85-95 [15] Faster R-CNN 89 [15] SSD 82 [16] Blob detector + count network + linear regression 91,3 Proposed system Yolo lite v3 94-99 To obtain responses from users, we carried out a survey of the application use feedback. Survey documents were disseminated through Google online survey covering several aspects, namely user- friendly, execution speed, accuracy, and benefits. Furthermore, each item of this question has a double answer option. Fig. 5 shows the feedback from users. Based on our survey, it was found that 95% agreed that the developed application was user-friendly, and only 5% of them disagreed. This is because we have developed an application to be able to process images from the user's smartphone camera. Users only need to take pictures of the oil palm harvest, and then our algorithm will process the image data automatically. Fig. 5. Users’ Feedback Users’ responses on the execution speed are quite good, with 70% stating that the application is fast, and 30% contended slow. Execution is carried out in the cloud, while the speed constraints are usually on uploading and downloading data transmission from the cloud, depending on the type of smartphone, signal strength, and internet user quota. From the research aspect, almost 85% of users responded that the image matching algorithm is accurate, and 15% of them expressed conversely. In general, this algorithm has been proven to be of good quality through performance tests so that the responses of 15% of users are probably due to the user's camera resolution or the user's poor photo technique. The last aspect is the benefit where 88% of users shared that the application entails many benefits, while 12% of them did not express the same idea. This shows that the developed application is acceptable for the users. 4. Conclusion The single-shot detector (YOLO) lite version 3 model has been successfully implemented in the smartphone application to speed up the process of harvesting palm oil for automatic counting. The applied model has high accuracy in detecting palm oil fruit. The network connection method used in the system more flexible because it can be used to calculate the amount of palm oil inside or outside the warehouse (garden area). Compared to the manual calculation, the proposed system can increase the efficiency of the harvest process with a low error rate. This could bring a precise number of fruit and reduce the embezzlement. In general, this application is plausible. It can be construed from the users’ positive responses to all survey questions concerning aspects of user-friendliness, accuracy, usefulness, and speed. Differences in users' responses to binary questions are due to differences in smartphone types and camera resolutions. Acknowledgment This work is supported by PNBP 2020 UM. % ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 81 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) References [1] A. C. Fitrianto, K. Tokimatsu, and M. Sufwandika, “Estimation the amount of oil palm trees production using remote sensing technique,” in IOP Conference Series: Earth and Environmental Science, 2017, vol. 98, no. 1, p. 12016. [2] L. K. Aguiar, D. C. Martinez, and S. M. Q. Caleman, “Consumer awareness of palm oil as an ingredient in food and non-food products,” J. food Prod. Mark., vol. 24, no. 3, pp. 297–310, 2018. [3] M. A. Shahputra and Z. Zen, “Positive and negative impacts of oil palm expansion in Indonesia and the prospect to achieve sustainable palm oil,” in IOP Conference Series Earth and Environmental Science, 2018, vol. 122, no. 1, p. 12008. [4] J. M. Ponce, A. Aquino, B. Millan, and J. M. Andújar, “Automatic counting and individual size and mass estimation of olive-fruits through computer vision techniques,” IEEE Access, vol. 7, no. May, pp. 59451– 59465, 2019. [5] A. Syal, D. Garg, and S. Sharma, “Apple fruit detection and counting using computer vision techniques,” in IEEE International Conference on Computational Intelligence and Computing Research, 2014, pp. 1– 6. [6] F. Kurtulmus, W. S. Lee, and A. Vardar, “Green citrus detection using ‘eigenfruit’, color and circular Gabor texture features under natural outdoor conditions,” Comput. Electron. Agric., vol. 78, no. 2, pp. 140–149, 2011. [7] Y. Çakır, M. Kırcı, E. O. Güneş, and B. B. Üstündağ, “Detection of oranges in outdoor conditions,” in Second International Conference on Agro-Geoinformatics (Agro-Geoinformatics), 2013, pp. 500–503. [8] K. Bresilla, G. D. Perulli, A. Boini, B. Morandi, L. Corelli Grappadelli, and L. Manfrini, “Single-shot convolution neural networks for real-time fruit detection within the tree,” Front. Plant Sci., vol. 10, no. May, pp. 1–12, 2019. [9] A. Payne, K. Walsh, P. Subedi, and D. Jarvis, “Estimating mango crop yield using image analysis using fruit at ‘stone hardening’ stage and night time imaging,” Comput. Electron. Agric., vol. 100, no. January, pp. 160–167, 2014. [10] W. C. Seng and S. H. Mirisaee, “A new method for fruits recognition system,” in 2009 International conference on electrical engineering and informatics, 2009, vol. 1, pp. 130–134. [11] Z. S. Pothen and S. Nuske, “Texture-based fruit detection via images using the smooth patterns on the fruit,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 5171–5176. [12] P. Nevavuori, N. Narra, and T. Lipping, “Crop yield prediction with deep convolutional neural networks,” Comput. Electron. Agric., vol. 163, no. April, p. 104859, 2019. [13] I. Sa, Z. Ge, F. Dayoub, B. Upcroft, T. Perez, and C. McCool, “Deepfruits: A fruit detection system using deep neural networks,” Sensors (Switzerland), vol. 16, no. 8, pp. 12–22, 2016. [14] M. Rahnemoonfar and C. Sheppard, “Deep count: Fruit counting based on deep simulated learning,” Sensors (Switzerland), vol. 17, no. 4, pp. 1–12, 2017. [15] H. Yu, S. Song, S. Ma, and R. O. Sinnott, “Estimating fruit crop yield through deep learning,” in BDCAT 2019 - Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, 2019, pp. 145–148. [16] S. W. Chen et al., “Counting apples and oranges with deep learning: A data-driven approach,” IEEE Robot. Autom. Lett., vol. 2, no. 2, pp. 781–788, 2017. [17] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in 32nd International Conference on Machine Learning, ICML 2015, 2015, vol. 3, pp. 1737– 1746. [18] L. Murmann, M. Gharbi, M. Aittala, and F. Durand, “A dataset of multi-illumination images in the wild,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, vol. 2019–Octob, pp. 4079–4088. 82 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 76-82 Aripriharta et.al (Smartphone for palm oil fruit counting to reduce embezzlement in harvesting season) [19] B. Arad, P. Kurtser, E. Barnea, B. Harel, Y. Edan, and O. Ben-Shahar, “Controlled lighting and illumination-independent target detection for real-time cost-efficient applications. The case study of sweet pepper robotic harvesting,” Sensors (Switzerland), vol. 19, no. 6, pp. 1–15, 2019. [20] M. Wang and Z. Shang, “Deep separable convolution neural network for illumination estimation,” ICAART 2019 - Proc. 11th Int. Conf. Agents Artif. Intell., vol. 2, no. 174, pp. 879–886, 2019.