International Journal of Interactive Mobile Technologies(iJIM) – eISSN: 1865-7923 – Vol 16 No 09 (2022) Paper—A New Classification Method for Drone-Based Crops in Smart Farming A New Classification Method for Drone-Based Crops in Smart Farming https://doi.org/10.3991/ijim.v16i09.30037 Bandar Al-Rami1, Khattab M. Ali Alheeti2(),Waleed M. Aldosari3, Saeed Matar Alshahrani4, Shahad Mahdi Al-Abrez2 1Dean of Faculty of Computer Studies, Arab open University, Riyadh, KSA 2College of Computer Sciences and Information Technology, Computer Networking Systems Department, University of Anbar, Ramadi, Iraq 3Department of Computer Engineering, Prince Sattam Bin Abdulaziz University, Al Kharj, Saudi Arabia 4Faculty of Computer Science and Information Technology, Saudi Electronic University, Riyadh, Saudi Arabia co.khattab.alheeti@uoanbar.edu.iq Abstract—During the past decades, smart farming became one of the most important revolutions in the agriculture industry. Smart farming makes use of different communication technologies and modern information sciences for in-creasing the quality and quantity of the product. On the other hand, drones showed a major potential for enhancing imagery systems and re-mote sensing usage for many different applications such as crop classification, crop health monitoring and weed management. In this paper, an intelligent method for clas- sifying crops is proposed to use a transfer learning approach based on a number of drone images. Moreover, the Convolution-al Neural Network (CNN) method is used as a classifier to improve efficiency for obtaining more accurate results in the training and testing phases. Various metrics are measured to evaluate the efficiency of the proposed model such as accuracy rate of detection, error rate and confusing matrix. It is found to be proven from the experimental results that the proposed method presents more efficient results with an accuracy detection rate of 92.93%. Keywords—crop classification, drone, transfer learning, smart farming 1 Introduction Over the past few years, the agriculture industry played an essential role in community development across the globe. The management system development is extremely significant for enhancing the outcome whilst monitoring the agriculture process usage for various environmental and economic purposes. Smart farming aims at integrating communications and information technologies in conventional farming operations [1]. Technologies such as the Internet of Things (IoT), Big Data Analytics (BDA), 164 http://www.i-jim.org https://doi.org/10.3991/ijim.v16i09.30037 mailto:co.khattab.alheeti@uoanbar.edu.iq Paper—A New Classification Method for Drone-Based Crops in Smart Farming Remote Sensing, Machine Learning (ML) and Unmanned Aerial Vehicles (UAVs) are favourable and can provide a modern development in several agriculture operations. Crop yields can be improved in smart farming for reducing costs and optimising pro- cess impacts (e.g., growth status, irrigation water, crop management, environmental conditions, greenhouse environment and soil status) [2][3]. The crop detection and classification topic are some of the conventional topics related to the remote sensing scientific community. Unmanned Aerial Vehicles (UAVs) have recently gained atten- tion by many researchers within the field [4][5]. In fact, they can provide local thematic information with temporal resolutions and higher spatial. The images of the UAV can lead to an increase in the number of classifiable targets and provide the recognition capability of different objects. When comparing the drone images with the satellite image [6]. Fig. 1. The classifications of the drone-based crops During the 2000s, the algorithms of Machine Learning such as the Conventional Neural Networks, Support Vector Machines and Random Forest have widely been used for classifying crops by using remote sensing information [7]. In this paper, Transfer Learning is applied as a crop detection model. The objective of this study is to design an accurate model with the ability to classify the crops efficiently depending on their drone images. The remainder of the paper is classified into five parts. In the second section, the related research is discussed briefly. The third section includes the transfer learning concept. In the fourth section, the proposed method is highlighted and explained in detail. The fifth section analyses the experimental results and discussion. Finally, the conclusion and gap of knowledge are drawn in the sixth section. 2 Related researches Recently, several researchers focus on classifying crops by drones in smart farm- ing using different methods. For instance, Geun-Ho K. and No-Wook P. (2019) explore the possibility of the texture information, which is based on the grey-level co-occurrence matrix (GLCM) for crops tabulation with machine learning methods such as Support Vector Machine (SVM) and Random Forest combined with time-series drone images [8]. They evaluate the effect of the combination of spectral information iJIM ‒ Vol. 16, No. 09, 2022 165 Paper—A New Classification Method for Drone-Based Crops in Smart Farming and texture on the performance of the crops classification when the input is found as multi-temporal images or a single drone image. The results show that using texture information is useful in the crops’ classification of high-resolution drone images. Autur N. et al. (2021) apply the transfer learning approach for performing crops’ classification in which high-resolution Red Green Blue-Unmanned Aerial Vehicles (RGB-UAV) images are used as an input for this operation [9]. The classification tasks use Convolutional Neural Networks (CNNs) that contain Google Net and VGG16. The proposed model detects the types of the crop accurately in Mozambique and Malawi datasets. The experimental results show that the number of frozen layers represents a serious parameter in the transfer learning approach and it is more effective than other approaches, which have also obtained more effective results on a few layers. Philip L. et al. (2017) implement a regular RGB camera that is installed on a light-weight drone for addressing crop classification such as typical weeds and sugar beets [10]. The researchers propose a classification system for detecting vegetation and extracting plant-tailored features. Their proposed system can obtain the estimation of the crops’ distribution in the fields. Robert Ch. et al. (2020) develop a deep learning algorithm to identify legumes, bananas, maize and other crops, which form the strategic crops in Rwandan’s agri- culture [11]. In particular, they use an RGB camera that is installed on a drone. The researchers employ a transfer learning approach and deep convolutional neural net- works in their study. It is found to be proven from their obtained results such as Banana and maize, can be detected effectively by the proposed model with high accuracy rates. M. Bah et al. (2018) apply an unsupervised training dataset with the Convolutional Neuronal Networks (CNNs) for proposing a new learning method that is fully auto- matic to detect crops and weeds by using drone images [12]. This method consists of three main stages. The first stage comprises crop lines detection, which is used to identify the interlined weeds. The second stage includes shaping the training dataset by using the automatically identified interlined weeds. In the last stage, the crops and weeds detection model are created by performing the CNNs on the dataset. The pro- posed model is performed in bean and spinach fields. M. Bah et al. (2020) use the Hough transform and the Convolutional Neural Net- works (CNNs) to produce a novel classification method for detecting crops by using drone images [1]. They called it the ‘CRowNet’ method. The proposed method contains a model that is created by Hough transform CNN and (S-SegNet). It is found to be proven from the experimental results that the method is more robust and effective in comparison with the conventional approaches with a detection rate of 93.58%. H. Hassanein et al. (2019) improve a low-cost drone RGB imagery system to propose a novel crop classification method [13], which consists of three funda- mental stages. The first stage includes the conversion of the RGB images into Hue- Saturation-Value (HSV) colour space, and subsequently, extracting the Hue image. The second stage includes generating various sections where each section possesses vari- ous orientation angles in the Hue images. The third stage includes generating a scan line with the same orientation angles. It is found to be proven from the experimental results that the proposed method can detect different types of crops accurately when evaluated in Canola fields. In [14], the researchers propose a semi-supervised learning model for crop classification. Red Green Blue (RGB) images are applied to evaluate 166 http://www.i-jim.org Paper—A New Classification Method for Drone-Based Crops in Smart Farming the proposed model. The results prove that the proposed model achieves an average accuracy of 90% when 80% of the training data is unlabelled. 3 Transfer learning approaches Despite the great success that is achieved by conventional machine learning tech- nologies, they still have some restrictions for many real-world scenarios [15]. The most important restriction is that the collection of the sufficiently collected training data is often unrealistic, expensive and time-consuming for several scenarios. Transfer learn- ing is a machine learning method that focuses on knowledge transferring across differ- ent domains where it can solve the above-indicated problem [16]. Fig. 2. The conventional machine learning approach versus the transfer learning approach Learning may be improved by transferring it along with three measures as follows [17]: • The time amount that it takes for learning the entire targeted tasks in which the knowledge is given to confront the required amount of time to learn from scratch. • The primary performance is realisable when using knowledge transfer within the targeted tasks that are confronted to the primary performance pertaining to the unlet- tered agent. • The conclusive performance level that is realised in the targeted tasks confront the conclusive level that is realised without being transferred. Transfer learning contains different techniques due to the existence of several types of conventional machine language algorithms [18]. The first strategy is called the ‘Inductive Transfer learning’ strategy in which the source task is unlike the targeted task, while the domains of the source and the target are just the same. The second strat- egy is called the ‘Transudative Transfer Learning’ strategy in which the source task is similar to the target task but the source domain and the target domain are various. The last strategy is the ‘Unsupervised Transfer Learning’ strategy in which the source task and the targeted task are ubiquitous, while the domains remain the same. The labelled data in this strategy is not obtainable from one of the mentioned domains. iJIM ‒ Vol. 16, No. 09, 2022 167 Paper—A New Classification Method for Drone-Based Crops in Smart Farming Fig. 3. Transfer learning techniques Figure 3 shows types of transfer learning approaches. 4 The proposed method In this paper, a new method is proposed to be trained on a base dataset; and to re-purpose it for many learning features or to transfer it to be trained through a bigger dataset. To obtain the training data, drone flight sites are selected for representing diver- sity in monocropping and intercropping. The crop labelling process uses GPS location capture and electronic survey instrument for visiting different agricultural areas. Crops are labelled remotely by using high-resolution drone images. The images are divided randomly into a model building training set and a model evaluation testing set. The benefit of the sampling process is to preserve the class ratios that are produced in the full labelled dataset. The proposed method consists of four phases (see Figure 4). Fig. 4. Block diagram of proposed method 168 http://www.i-jim.org Paper—A New Classification Method for Drone-Based Crops in Smart Farming These phases are comprised of: • Image acquisition: the labelled dataset is created and analysed in this phase. It con- tains the images that are captured by a drone camera. • Pretrained model: it is a model that is trained on datasets based on the use of a trans- fer learning approach for extracting features related to any image. • Feature extraction: during this phase, the images are represented from new datasets by using pre-trained features. • Crop classification: this phase includes the detection of a crop type that is present in the input image. The probabilities of the found class are presented by the output results. 5 Experiment results and discussion In this research, the performance of the classification results is illustrated by using different metrics such as confused matrix and accuracy rate of detection. The confusion matrix is defined by four measures, which comprise: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). The measures are calculated as follows [19]: TP TP TP FNRate sensitivity( ) � � (1) TN TN TN FPRate specificity( ) � � (2) FN FN FN TPRate sensitivity( )1� � � (3) FP FP FP TNRate specificity( )1� � � (4) The accuracy rate is calculated as follows [20]: Accuracy = Number of correctly classified patterns Total number off patterns (5) In this paper, CNN is used as a classifier for more accurate results in the training and testing phase. Some parameters of the CNN classifier are used for training as follows: Primary Training Parameters: • Epochs no. = 500; • TrainParam.lr = 0.1–6; • Goal = 0; • Min_grad = 0.1–14 iJIM ‒ Vol. 16, No. 09, 2022 169 Paper—A New Classification Method for Drone-Based Crops in Smart Farming A neural network is created with four layers, which are the input layer (13 neurons), two hidden layers (nine neurons and five neurons) and the output layer (six neurons— Healthy Leaf, Cercospora Leaf Spot, Bacterial Blight, Anthracnose, Alternaria Alter- nate and Unknown). The results of the accuracy rate and confusing matrix are presented for four rounds in Tables 1–3, with a total training accuracy of 92.93%. 1) First Round: Testing Accuracy of 90.13%: Table 1. Performance metrics_1 Detection Class Accuracy Rate (%) Healthy Leaf 0 Alternaria_Alternata 94.96 Anthracnose 85.39 Bacterial_Blight 75.91 Cercospora_Leaf_Spot 87.34 Unknown 0 Confused Matrix’s Rate (%) TP 100 TN 100 FP 0 FN 0 Error Rate 9.86 The performance metrics of the first-round show that the rates of the TP and TN reach 100% (see Table 2). 1) Second Round: Testing Accuracy of 89.66%: Table 2. Performance metrics_2 Detection Class Accuracy Rate (%) Healthy Leaf 91% Alternaria_Alternata 87.94 Anthracnose 91.33 Bacterial_Blight 73.70 Cercospora_Leaf_Spot 79.70 Unknown 0 Confused Matrix’s Rate (%) TP 92 TN 100 FP 0 FN 8 Error Rate 10.33 As shown in Table 2, the TN rate is greater than 100%. 170 http://www.i-jim.org Paper—A New Classification Method for Drone-Based Crops in Smart Farming 2) Third Round: Testing Accuracy of 91.20% Table 3. Performance metrics_3 Detection Class Accuracy Rate (%) Healthy Leaf 98.32 Alternaria_Alternata 100 Anthracnose 70.18 Bacterial_Blight 99.67 Cercospora_Leaf_Spot 100 Unknown 0 Confused Matrix’s Rate (%) TP 98.2 TN 96.54 FP 3.46 FN 1.8 Error Rate 8.80 3) Fourth Round: Testing Accuracy of 90.46% Table 4. Performance metrics_4 Detection Class Accuracy Rate Healthy Leaf 92.67 Alternaria_Alternata 89.75 Anthracnose 76.34 Bacterial_Blight 100 Cercospora_Leaf_Spot 86.83 Unknown 0 Confused Matrix’s Rate (%) TP 94.1 TN 97.79 FP 2.21 FN 5.9 Error Rate 9.53 According to Table 4, it is noticed that the proposed detection system can detect various types of crops effectively. 6 Conclusion & future work UAVs offer the possibility to monitor every farm for every plant premise, which thus, can diminish the measure of herbicides and pesticides that are applied. A focal data for the farmer just as for independent farming robots is the information about the iJIM ‒ Vol. 16, No. 09, 2022 171 Paper—A New Classification Method for Drone-Based Crops in Smart Farming kind and appropriation of the weeds in the field. In such a manner, UAVs offer efficient review capacities for a minimal price. The transfer learning method is applied to solve different challenging tasks. In this paper, an intelligent method is produced to classify multiple crops by using a transfer learning approach based on existing drone images. Furthermore, the CNN approach is used as a classifier to achieve more accurate results in the training and testing phases. Different metrics are measured to evaluate the effi- ciency of the proposed model such as accuracy rates of detection, error rates and con- fusing matrices. It is found to be proven from the obtained results that the proposed model presents more effective results where the total accuracy detection rate reaches 92.93%. In future research, the produced detection system can have the potential to be tested along with other suitable datasets. 7 Acknowledgment The author(s) would like to thank the Solution Makers Institute for Training (Ora-cle), Saudi Arabia for supporting this study. 8 References [1] M. D. Bah, A. Hafiane, and R. Canals, “CRowNet: Deep Network for Crop Row Detec- tion in UAV Images,” IEEE Access, vol. 8, pp. 5189–5200, 2020. https://doi.org/10.1109/ ACCESS.2019.2960873 [2] P. Diamantoulakis et al., “Internet of Things (IoT) and Agricultural Unmanned Aerial Vehi- cles (UAVs) in Smart Farming: A Comprehensive Review,” Internet of Things, p. 100187, 2020. https://doi.org/10.1016/j.iot.2020.100187 [3] D. C. Tsouros, S. Bibi, and P. G. Sarigiannidis, “A review on UAV-based applications for precision agriculture,” Inf., vol. 10, no. 11, 2019. https://doi.org/10.3390/info10110349 [4] P. K. R. Maddikunta, S. Hakak, M. Alazab, and S. Bhattacharya, “Unmanned Aerial Vehicles in Smart Agriculture: Applications, Requirements and Challenges,” 2020. [5] M. Esposito, M. Crimaldi, V. Cirillo, F. Sarghini, and A. Maggio, “Drone and sensor technol- ogy for sustainable weed management: a review,” Chem. Biol. Technol. Agric. Vol., vol. 8, 2021. https://doi.org/10.1186/s40538-021-00217-8 [6] A. Mora, T. Santos, S. Lukasik, J. Silva, A. Falcao, J. Fonseca, and R. Ribeiro, “Land cover classification from multispectral data using computational intelligence tools: A comparative study,” Inf. 2017, vol. 8, p. 147, 2017. https://doi.org/10.3390/info8040147 [7] U. S. Panday, A. K. Pratihast, J. Aryal, and R. B. Kayastha, “A review on drone-based data solutions for cereal crops,” Drones 2020, vol. 4(3), 2020. https://doi.org/10.3390/ drones4030041 [8] G. Kwak, “Applied sciences Impact of Texture Information on Crop Classification with Machine Learning and UAV Images,” 2019. https://doi.org/10.3390/app9040643 [9] A. Nowakowski et al., “International Journal of Applied Earth Observations and Geoinforma- tion Crop type mapping by using transfer learning,” Int. J. Appl. Earth Obs. Geoinf., vol. 98, no. January, p. 102313, 2021. https://doi.org/10.1016/j.jag.2021.102313 [10] P. Lottes and J. Pfeifer, “UAV-Based Crop and Weed Classification for Smart Farming,” Conf. IEEE Int. Conf. Robot. Autom. (ICRA)At Singapore, 2017. https://doi.org/10.1109/ ICRA.2017.7989347 172 http://www.i-jim.org https://doi.org/10.1109/ACCESS.2019.2960873 https://doi.org/10.1109/ACCESS.2019.2960873 https://doi.org/10.1016/j.iot.2020.100187 https://doi.org/10.3390/info10110349 https://doi.org/10.1186/s40538-021-00217-8 https://doi.org/10.3390/info8040147 https://doi.org/10.3390/drones4030041 https://doi.org/10.3390/drones4030041 https://doi.org/10.3390/app9040643 https://doi.org/10.1016/j.jag.2021.102313 https://doi.org/10.1109/ICRA.2017.7989347 https://doi.org/10.1109/ICRA.2017.7989347 Paper—A New Classification Method for Drone-Based Crops in Smart Farming [11] R. Chew, J. Rineer, R. Beach, M. O’Neil, N. Ujeneza, D. Lapidus, T. Miano, M. Hegarty- Craver, J. Polly, and D. S. Temple, “Deep neural networks and transfer learning for food crop identification in UAV images,” 2020. [12] M. D. Bah, A. Hafiane, and R. Canals, “Deep Learning with unsupervised data labeling for weeds detection on UAV images,” arXiv. pp. 1–11, 2018. https://doi.org/10.20944/pre- prints201809.0088.v1 [13] M. Hassanein, M. Khedr, and N. El-Sheimy, “Crop row detection procedure using low-cost uav imagery system,” Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. – ISPRS Arch., vol. 42, no. 2/W13, pp. 349–356, 2019. https://doi.org/10.5194/ isprs-archives-XLII-2-W13-349-2019 [14] Shahbaz Khan, Muhammad Tufail, Muhammad Tahir Khan, Zubair Ahmad Khan, Javaid Iqbal, and Mansoor Alam, “A novel semi-supervised framework for UAV based crop/weed classification,” PLoS One, 2021. [15] F. Zhuang et al., “A comprehensive survey on transfer learning,” Proc. IEEE, vol. 109, no. 1, pp. 43–76, 2020. https://doi.org/10.1109/JPROC.2020.3004555 [16] B. U. C. McCool and T. Perez, “Mixtures of lightweight deep convolutional neural networks: Applied to agricultural robotics,” IEEE Robot. Autom. Lett., vol. 2, no. 3, pp. 1344–1351, 2017. https://doi.org/10.1109/LRA.2017.2667039 [17] D. Sarkar, “A comprehensive hands-on guide to transfer learning with real-world applica- tions in deep learning,” Medium, vol. 11, no. 14. pp. 1–83, 2018. [18] Andrzej Brodzicki, Michal Piekarski, Dariusz Kucharski, Joanna Jaworek-Korjakowska, and Marek Gorgon, “Transfer learning methods as a new approach in computer vision tasks with small datasets,” Foundations of computing and decision sciences, vol. 45, 2020. https:// doi.org/10.2478/fcds-2020-0010 [19] K. M. A. Alheeti, A. Alaloosy, H. Khalaf, A. Alzahrani, and D. Al_Dosary, “An optimal dis- tribution of RSU for improving self-driving vehicle connectivity”, Computers, Materials & Continua. http://dx.doi.org/10.32604/cmc.2022.019773 [20] M. S. A. R. Khattab, M. Ali Alheeti, W. Venus, “The effect of fuzzification on neural net- works intrusion detection system,” 2009 4th IEEE Conf. Ind. Electron. Appl. ICIEA 2009, pp. 1236–1241, 2009. 9 Authors Dr. Bandar Ali Al-Rami Al-Ghamdi, holds a PhD from Universite de Reims Champagne-Ardenne in 2015, Reims, France, the M.Sc. degree in Information Tech- nology from De Montfort University, Leicester, United Kingdom, in 2008 and the B.Cs. degree in Computer Sciences from the University of King Abdul-Aziz Univer- sity, Jeddah, Saudi Arabia, in 2003. Currently he is a Dean of Faculty of Computer Studies at Arab Open University, Riyadh, KSA. His research interests are Sensor Networks, Distributed Systems, eHealth Systems, Networking, Testing, Verification, Software Engineering and Real-Time Systems. He has multiple publications in national and international sources. Khattab M. Ali Alheeti is an Associate Professor at the Department of Computer Networking Systems, College of CS IT – University of Anbar, Iraq, where he has been a faculty member since 2009. From 2018 – now, he was also Head of Computer Net- working Systems Department. Alheeti graduated with a first-class honours Msc. Sci. degree in CS IT from AL Byte University, Jordan, in 2008, and an PhD. in CS IT iJIM ‒ Vol. 16, No. 09, 2022 173 https://doi.org/10.20944/preprints201809.0088.v1 https://doi.org/10.20944/preprints201809.0088.v1 https://doi.org/10.5194/isprs-archives-XLII-2-W13-349-2019 https://doi.org/10.5194/isprs-archives-XLII-2-W13-349-2019 https://doi.org/10.1109/JPROC.2020.3004555 https://doi.org/10.1109/LRA.2017.2667039 https://doi.org/10.2478/fcds-2020-0010 https://doi.org/10.2478/fcds-2020-0010 http://dx.doi.org/10.32604/cmc.2022.019773 Paper—A New Classification Method for Drone-Based Crops in Smart Farming from Essex University, UK in 2017. Him research interests are primarily in the area of self-driving vehicles, wireless communications and networks as well as security, particularly in neuro-signal processing and intrusion detection, where he is the author/ co-author of over 100 research publications. He can be contacted at email: co.khattab. alheeti@uoanbar.edu.iq. Waleed Aldosari is now an assistant professor of the Department of Computer Engineering, Prince Sattam Bin Abdulaziz University, Saudi Arabia, and the focus of his current mainly lies in wireless security Dr. Aldosari received the B.Sc. and M.S. degree in computer engineering in 2010 and 2015 respectively and received a Ph.D. degree in Electrical and Computer engineering from the University of Oakland, USA in 2019. From March 2020 to March 2021, he worked as a postdoc at King Abdulaziz University, where he had designed an efficient algorithm for localizing UAV jamming, his research interests included wireless sensor networks, anti-jamming, Drone Commu- nication, localization, wireless security, and physical security. Saeed Matar Alshahrani Received MSc degree in Information Technology from Sacred Heart University USA (2016). He works at the Faculty of Computer Science and Information Technology, Saudi Electronic University, Saudi Arabia as Lecturer. His research interests are: e-portfolio System, Machine learning. He can be contacted at email: alshahrani.s@seu.edu.sa. Shahad Mahdi Al-Abrez, Anbar, holds a master’s degree in computer science from the College of Computer Science and Information Technology, Anbar University, my interests are in the field of networks and information security. Article submitted 2022-02-07. Resubmitted 2022-03-10. Final acceptance 2022-03-11. Final version published as submitted by the authors. 174 http://www.i-jim.org mailto:co.khattab.alheeti@uoanbar.edu.iq mailto:co.khattab.alheeti@uoanbar.edu.iq mailto:alshahrani.s@seu.edu.sa