TX_1~ABS:AT/ADD:TX_2~ABS:AT 54 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 ReseaRch aRticle Converting a DJI Tello Quadcopter into a Face-follower Machine using the Haar Cascade with PID Controller Shalaw M. Abdallah Department of Software Engineering, Koya University, Kurdistan Region, Iraq ABSTRACT Drones have been frequently used for photography in recent years at significantly cheaper rates. However, the most modern drones are exceedingly error-prone and require precise manual control to take high-quality photos or films. We suggest using the AI method of Haar cascades with a PID controller to give drones vision, enabling them to do autonomous tracking and detection. This project aims to improve photography fields. The proposed system tries to detect the face and track the person’s movements. This system will help photographers and journalists upgrade their work, even if it is used in surveillance and the military. The algorithm’s results show that the DJI Tello tiny drone’s camera is capable of detecting and tracking faces. The micro drone was picked since it is lightweight and compact, making its use safe and enabling testing to take place inside. In addition, the DJI Tello may be easily programmed using Python. The position of the drone is contrasted with the set point in the center of the image to identify errors, allowing control signals for calculating forward/backward, right/left, and yaw movements. The proposed system results show that the drone can detect and track the face very well, and the PID values are stable. Keywords: Backward/forward, DJI Tello, Haar Cascade, PID controller, right/left, Tello drone INTRODUCTION Today’s drones are packed with cameras and have a lot of possible uses for a range of industrial applications, among them photography, surveillance, etc. Drones must be packed with novel computer vision to be widely utilized and to further cut down on their costs. To capture important elements in a picture while using photography or surveillance, object identification, and tracking are crucial. In computer vision, object detection and tracking are common issues.[1] The employment of drones in considerably more commercial fields, including photography, cinematography, surveillance, and even delivery systems, has been made possible as a result. Thus, becoming proficient in programming drones to carry out various duties is a skill that is in high demand. Drones come in a variety of sizes and shapes. A popular term you might have heard for a drone is quadcopter. This points out a drone with four propellers or motors. Likewise, a hexacopter has six propellers, and an octocopter has eight propellers. A drone’s frame, motors, propellers, global positioning system (GPS), power distribution board, flight controller, battery, bluetooth receiver, camera, video transmitter, and a few sensors will make up its basic components; everything is held together by the body. The system that generates lift and enables the drone to fly can be thought of as having motors as its engine.[2,3] A drone uses two different types of motors. There are two different kinds of motors used in drones, both brushed and non-brushed. Brushless motors are more expensive but have a superior cost-to-power ratio than brushed motors, which are straightforward and inexpensive. The lift force is produced by the propellers as soon as the motors begin to rotate. Any given drone will have two different sorts of propellers, one that rotates in the clockwise direction and the other in the anticlockwise direction. This is due to the fact that the drone will begin to rotate if all of the propellers are spinning in the same direction. The drone does not rotate; however, because they can cancel each other out because we have half of them rotating in one direction and the other half rotating in the opposite direction. Multiple blade pairs are possible for propellers. The thrust produced by having more blades increases, but their efficiency decreases.[4] The speed of the motors is managed by the electric motor controller. They transform the battery’s direct current signal Corresponding Author: Shalaw M. Abdallah, Department of Software Engineering, Koya University, Kurdistan Region, Iraq. E-mail: shalaw.mshir@gmail.com Received: October 02, 2023 Accepted: November 12, 2023 Published: December 20, 2023 DOI: 10.24086/cuesj.v7n2y2023.pp54-59 Copyright © 2023 Shalaw M. Abdallah. This is an open-access article distributed under the Creative Commons Attribution License. Cihan University-Erbil Scientific Journal (CUESJ) https://creativecommons.org/licenses/by-nc-nd/4.0/ Abdallah: DJI Tello Quad: Face-Following system with Haar Cascade and PID Control 55 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 into an alternating current signal for the motor. A drone has several sensors attached to it. For instance, the pressure sensor calculates the drone’s altitude or its distance from the ground. The drones are using the Global Positioning System (GPS) and an inertial measurement unit (IMU). The IMU is an electrical instrument that can be used to detect acceleration and angular motion. It integrates the sensors from gyroscopes, magnetometers, and accelerometers into a single sensor. With the aid of a time derivative, the accelerometer can also be utilized to calculate metrics like velocity and distance. Signals are received using a GPS module. Radio receivers in the GPS module receive signals from a global network of orbiting satellites. The module can calculate its position, velocity, and time thanks to these messages. Access to more signals and systems improves location accuracy, which benefits all of the drone’s GPS-related operations.[5,6] In spite of the fact that a drone contains four or more motors to accomplish all movements, moving one is actually pretty easy and intuitive. The drone can spin in one direction and translate in three other directions thanks to its 4° of freedom of motion. A quadcopter has four propellers, four of which revolve clockwise and two of which rotate counterclockwise, as was previously described. By creating a zero angular momentum as a result, the drone can remain still rather than rotating in one direction.[7] Lets have a look at the translation movements. First, the system will run all the motors at the same speed. That will generate the lift greater than the weight of the drone. The drone will rise as a result of this. The initial translational movement is this: If the proposed system wants to descend, it will slow down each motor so that the lift is less than the drone’s weight. The proposed system will adjust the motor’s speed if it wants to hover in the air so that the lift is equivalent to the drone’s weight. The drone will be able to hover over the ground as a result. Moving on to the second translation, the system will slow down the left motors and speed up the right motors if it wishes to move to the left. The system shall raise the speed of the left motors while decreasing the speed of the right motors to move to the right. Similar translations are possible in the third direction, which is forward and backward motion. The drone will advance by reducing the front motors’ speed and increasing the back motors’ speed. The system can change the speed of the front motors while reducing the speed of the back motors to move backward. The system can utilize this knowledge to rotate our drone because two motors rotate clockwise and the other two rotate anticlockwise. The system will make the motors moving anticlockwise faster and the motors moving clockwise slower to rotate the drone clockwise. The clockwise motors’ speed will increase, and the anticlockwise motors’ speed will decrease if the system wants to rotate in the other direction.[8] In addition, this research proposal will help photographers and journalists prepare their works in a good way. The system will let the drone follow the face of the person to record or take images. We chose the face because the face is the most important part of the body. By focusing on the face, photographers and journalists can capture the emotions and expressions that convey powerful storytelling elements. This innovative feature will enable them to create visually compelling content that resonates with their audience on a deeper level. Furthermore, by automating the tracking process, the system will provide photographers and journalists with more freedom to concentrate on framing their shots and capturing the perfect moment. RELATED WORK The issue of facial recognition, detection, and tracking from a drone has been addressed more recently, as advancements in technology have made it possible for drones to incorporate facial recognition software. Many studies have been published in this field and try to solve this problem in fields such as photography or surveillance. These studies aim to improve the accuracy and efficiency of facial recognition algorithms when applied to drone technology. In addition, researchers are exploring the ethical implications and privacy concerns associated with the use of facial recognition on drones, further complicating the development and implementation of this technology. The study[9] proposes a face recognition-based drone for tracking illicit and missing individuals using Haar Cascade and Local Binary Pattern Histogram algorithms. The drone achieved 98 percent accuracy in identifying individuals, making it a valuable tool for security, self-confirmation, and disaster relief in rural regions. The paper[10] presents a Haar-Cascade-based classifier for face detection and recognition on a 34.7 µg Unmanned Aerial Vehicle (UAV), using OpenCV Python. The face detection process is realtime, using an indoor FPV camera. The experiment reveals that the angle of the face is crucial, especially for non-open faces, and the optimal angle is determined by the camera’s slope. This paper[11] proposes a solution using a face recognition- based UAV for identifying criminals, missing people, and civilians. The system uses a camera linked to Face Recognition software and a wireless remote control. The drone’s accuracy is 98.6% when the camera angle is 37°, aiding police investigations in search and rescue. This paper[12] proposes a facial recognition system using a local binary patterns histogram face recognizer mounted on drone technology for identifying criminals in crowded areas. The system can be used as a surveillance drone, covering more areas than a stationary system. It tags the person and transmits the image and location coordinates to authorities using a mounted GPS, achieving an accuracy of approximately 89.1%. DJI TELLO DRONE The drone we will be utilizing for this research is the Tello, which can be seen in Figure 1. Although this drone is recommended for use with this course, the same concepts can be applied to other drones. In truth, only 20% of this training is particular to the Tello drone, while 80% of this research is applicable to any other drones. A small drone called the Tello is fitted with a camera that can record 720p video at 30 frames per second. It is Abdallah: DJI Tello Quad: Face-Following system with Haar Cascade and PID Control 56 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 manufactured by Rise and has DJI and internal flight control technology, as well as internal lighting and sensors. Because it has various safety features, it is very simple and safe to use indoors[13]. The nicest feature about this drone for learning drone programming is that it is programmable, which means it is open source and we can use computer vision technologies on it. The Tello may be operated using a mobile application that connects to a Wi-Fi network and is compatible with both Apple and Android devices. It contains a lot of features and is quite user-friendly. MATRIALS AND METHODS The additional functions found on the Mini Tello drone are its main benefits such as the VPS and an onboard camera, which are the key advantages of the Mini Tello drone. This small quadcopter hovers in a quiet posture thanks to these capabilities and an advanced control system. This small drone’s inbuilt camera is capable of taking 5-megapixel images and streaming 720p video in real-time. With a maximum distance of around 100 m, 13 min is the longest flight possible. This drone features a failsafe protection mechanism; it implies that the drone will safely land even if the connection is lost. The drone’s other safety feature is that the engine will stop rotating if it collides with an item hard enough.[14] This drone can programmability with Python and Swift. Using the Djitellopy library, we can have access to the sensor data on the DJI Tello and execute instructions. The materials used in the research are shown in Table 1. A DJI Tello drone with a laptop attached can communicate over a Wi-Fi network. Using a computer vision technique that is described in the methodology section, the DJI Tello camera takes a picture that is then delivered to a laptop for processing. Table 2 shows the detailed specifications of the computer utilized in this research. There are a lot of amazing projects that can be done with Djitellopy library that made the behavior like an AI machine for instance (Face tracking, Object detection, Hand gesture, etc), for this project we will focus on face detection and tracking. For face detection, there are a lot of amazing methods could be used with great results, but need large Figure 1: Mini Quadcopter DJI Tello Table 2: Materials Materials Name CPU Intel i7-7200U (2C 4T, Max 3.10 Ghz) RAM DDR4 12GB (8GB+4GB) Connectivity WiFi 2.4Ghz Webcam 760p OS Ubuntu 20.04.4 LTS Table 1: Materials Materials Name Version DJI Drone Mini Tello Mini PL Python 3.6 v library DJITelloPy 2.4 v IDE PyCharm 21 computation. Therefore, the choice of the Haar cascade method was influenced by the computing system’s limitations. The collection of Haar features is the initial stage in the Haar cascade method. A computation known as a “Haar feature” is made on the neighboring square area at a certain spot in a detection window. To calculate the difference between the summed findings, the pixel intensities in each segment must first be added together. Edge features, line features, and four-rectangle features are a few examples of haar features. Because there are fewer operations required, an integral image process is required because this feature will have issues when used with huge images. Integral images are designed to accelerate the computation of haar features. Overall, an integral image is one in which the values of the top-left and left-most pixels are added together to determine the value of each individual pixel. It should be observed that practically every feature employed performs poorly when trying to recognize things. Therefore, to detect an object, we need some features that match it. The best Haar feature to utilize in this situation can be chosen by using Adaboost. By combining a number of small (weak) classifiers, the AdaBoost algorithm creates powerful classifiers. By repositioning the window over the input image and computing the Haar feature for each subsection, weak classification results are produced. This variation is contrasted with the researched threshold that distinguishes objects from non-objects. Numerous Haar features are needed to create a strong classifier because this is a “weak classifier” in terms of accuracy. The Haar cascade method combines cascades of classifiers to increase detection speed by focusing on likely areas in an image. The algorithm consists of three levels, each producing a sub-image believed to be unrelated to the object being detected, making it easier to distinguish between them. Figure 2 simplifies the Haar algorithm process.[14-16] The ideal method for tracking the front face is to use the Djitellopy package, which was built in Python by Damià Fuentes. Let’s start with the basics: When we want to move the drone and follow our face, we need two things: forward and backward motion, as well as rotational motion. The concept of forward and backward is like that: Abdallah: DJI Tello Quad: Face-Following system with Haar Cascade and PID Control 57 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 1. If the face looking small, so it is mean we need to move forward. 2. If the face looking quit big, it is mean we need to backward. Let us take a look on Figure 3, in Figure 2, the area (A) is too far and the area (C) is too close, the red area which is (B) is the area that the drone will never move to it, the person must be in this red area, otherwise the drone cannot detect the face. Use If the person moves to Zone A (backward), the drone will begin moving and counting the movements; the drone will then compensate and move forward until it reaches the red zone; if the person moves forward, the drone will begin moving backward and counting the movements; the drone will then compensate and move forward until it reaches the red zone; if the person moves forward, the drone will begin moving backward and counting the movements; the drone will then stop until it reaches the red zone. Then consider the concept of rotation: if the person begins to rotate, the goal is for the person to always be in the center of the image, as shown in Figure 4, the person begins to rotate, therefore, the idea is that the drone will begin to rotate to the left until the person reaches the center. The drone will start rotating to the left as shown in the Figure 5 until the person is located in the middle. However, there is a problem here, for example, if the drone rotates at 20 cm/s to reach the destination, the problem is that the drone cannot stop right away, the momentum will cause move a little bit to forward, picture a fan, if you turn off the fan, it will keep rotating and eventually stop, the same thing will happen with the Tello, this is called overshoot. We can address this problem by reducing the speed when Tello reaches the goal; once we get close to the destination, the Tello’s speed will drop to 10cm/s, then 5cm/s, and finally 0cm/s; the person will be in the center, and all will be well. PID is the name for this approach. PID was utilized to reduce overshooting in this case.[17,18] The following (1) defines Proportional-Integral-Derivative: ( ) = + + = + +∫ 1 0 ( ) * * de t u t p I D Kp Err ki Err dt kd dt (1) 1. The disparity among the set point and the process variable determines the proportional component. Enhancing the proportion will often quicken the response time of the control system. The process component will start to oscillate if the proportionate is too high. Further increasing the proportionate will result in bigger oscillations, an unstable system, and possibly even uncontrollable oscillations. 2. The quantity and duration of the error values are represented by the integral error, which takes the integral Figure 2: Steps of Haar cascade algorithm Figure 3: Drone Zones Figure 4: Yaw angle of the errors that the control loop records. The integral can determine both the current error value and previous error values since it takes time into account, while the proportional term cannot. A higher integration factor reduces steady-state error, or the output value’s ringing around the target value. As a result, the system can stabilize more rapidly and accomplish its objective. Higher integral coefficients, however, may cause more overshoot when first reaching the target. 3. Increasing the derivative time parameter will speed up the overall responsiveness of the control system and make the control system respond to changes in the error term more quickly. Due to the relatively high sensitivity of the derivative response to noise in the process variable signal, the majority of practical control systems employ very small derivative. The derivative response can lead to control system instability if the sensor feedback Abdallah: DJI Tello Quad: Face-Following system with Haar Cascade and PID Control 58 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 Figure 5: Rotating the drone signal is noisy or if the control loop rate is excessively slow.[19,20] Moreover, here the error is the difference between the face of the person and the bound box of the Haar algorithm. The PID algorithm tries to reduce the distance between these two points by founding a stable point that makes the done be stable enough to make it move to roll, pitch, and yaw to follow the face correctly. IMPLEMENTATION AND RESULTS Figure 6 depicts the flowchart used during the investigation. In general, it is an iterative method of face tracking and signal calculation for controlling speed movements. The AI drone is capable of moving in any direction with the right PID parameters, when a human stop moving, the AI drone can precisely track their face and stop in front of them.[21] You can view the experimental result throw below link: https://www.youtube.com/watch?v=COU7tcpg2qs In this video, a quadcopter follows a face as it rotates and goes in different directions, starting off to the right and then to the left. Then, the human moves forward and the quadcopter moves backwards; at that point, you can clearly see the yaw rotation. Then, the face moves in close proximity to the drone, and the drone follows the face before stopping. The method used in this study involves fixing a point in the middle of the image with a bounding box of a particular size. The quadcopter’s position error rate against the object’s face is calculated by comparing the center bounding box to the findings of object identification bounding boxes. The result of calculating PID parameters is shown in Table 3. The proposed system has three movements: Left/ right (roll), forward/backward (pitch), and yaw. At these points, there was stability for the drone movements. The PID controller was managed manually by Betaflight. Betaflight is a flight controller application executed to fly multirotor craft and fixed-wing craft. It is open-source software that is highly customizable and widely used in the drone racing community. Betaflight offers advanced features such as PID tuning, making it a popular choice among drone enthusiasts for its performance and flexibility.[22] Figure 7 shows the step response function of the PID, and we can clearly see that the drone movements are stable Figure 6: Face tracking flowchart Figure 7: The PID values graph for the drone roll, pitch, and yaw movements Abdallah: DJI Tello Quad: Face-Following system with Haar Cascade and PID Control 59 http://journals.cihanuniversity.edu.iq/index.php/cuesj CUESJ 2023, 7 (2): 54-59 at the given points. Note that these points can be changed; the system can find more stable points, but for this research, these points will be essential. CONCLUSION We described the tracking and detecting of the face in this article, which seeks to keep a face within a certain distance of the drone and roughly in the middle of the camera stream. To do this, three types of movements are used: Yaw rotation, left/ right, and forward/backward. In addition, we covered how to use the Haar cascade and how to control the drone movements with the PID controller. We can add more movements in the future, such as up and down, and work more on the stable points. REFERENCES 1. S. Han, W. Shen and Z. Liu. Deep Drone: Object Detection and Tracking for Smart Drones on Embedded System, 2016. Available from: https://web.stanford.edu/class/cs231a/prev_ projects_2016/deepdrone-object 2_.pdf 2. T. Lee, S. Mckeever and J. Courtney. Flying free: A research overview of deep learning in drone navigation autonomy. Drones, vol. 52, p. 52, 2021. 3. K. Kakaes. From orville wright to september 11: What the history of drone technology says about its future. In: Drone Wars: Transforming Conflict, Law, and Policy, Cambridge University Press, Cambridge, pp359-387, 2015. 4. J. B. Song and S. K. Byun. Design and control of an omnidirectional mobile robot with steerable omnidirectional wheels. In: Mobile Robotics, Moving Intelligence. IntechOpen, London, 2006. 5. L. M. Sukma. Tracking Object based on GPS and IMU Sensor. In: 2018 5th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE). IEEE, United States, 2018. 6. A. Laurell, E. Karlsson and Y. Naqqar. GPS and IMU Sensor Fusion to Improve Velocity Accuracy. Uppsala Universitet, Sweden, 2022. 7. C. Ye, S. Ma and L. Hui. An omnidirectional mobile robot. Science China Information Sciences, vol. 54, pp. 2631-2638, 2011. 8. F. G. Francois and S. M. Killough. A new family of omnidirectional and holonomic wheeled platforms for mobile robots. IEEE Transactions on Robotics and Automation, vol. 10, no. 4, pp. 480- 489, 1994. 9. K. G. Shanthi, S. S. Vidhya, K. Vishakha, S. Subiksha, K. K. Srija and R. S. Mamtha. Haar cascade face detection and local binary pattern histogram face recognition based drone. In: 2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS). vol. 1. pp. 295-298, 2022. 10. A. P. Atmaja, S. B. Setyawan, L. D. Setia, S. V. Yulianto, B. Winarno and T. Lestariningsih. Face recognition system using micro unmanned aerial vehicle. Journal of Physics: Conference Series. vol. 1845, no. 1, p. 012043, 2021. 11. K. G. Shanthi and P. Sivalakshmi. Smart drone with real time face recognition. Materials Today: Proceedings, vol. 80, pp. 3212-321, 2023. 12. L. Wang and A. A. Siddique. Facial recognition system using LBPH face recognizer for anti-theft and surveillance application based on drone technology. Measurement and Control. vol. 53, no. (7-8), pp. 1070-1077, 2020. 13. B. Vergouw, H. Nagel, G. Bondt and B. Custers. Drone technology: Types, payloads, applications, frequency spectrum issues and future developments. In: The Future of Drone Use: Opportunities and Threats from Ethical and Legal Perspectives, Springer, Germany, pp. 21-45, 2016. 14. A. S. Priambodo, F. Arifin, A. Nasuha and A. Winursito. Face tracking for flying robot quadcopter based on haar cascade classifier and PID COntroller. Journal of Physics: Conference Series. vol. 2111, no. 1, p. 012046, 2021. 15. A. Kasinski and S. Adam. The architecture and performance of the face and eyes detection system based on the Haar cascade classifiers. Pattern Analysis and Applications, vol. 13, no. pp. 197- 211, 2010. 16. R. Lienhart, A. Kuranov and V. Pisarevsky. Empirical analysis of detection cascades of boosted classifiers for rapid object detection. In: Pattern Recognition: 25th DAGM Symposium, Magdeburg, Germany, September 10-12, 2003. Proceedings 25. Springer, Berlin, Heidelberg, 2003. 17. A. L. Salih, M. Moghavvemi, M. Haf and K. Gaeid. Flight PID controller design for a UAV quadrotor. Scientific Research and Essays, vol. 5, no. 23, pp. 3660-3667, 2010. 18. W. Raza, A. Osman, F. Ferrini and F. De Natale. Energy-efficient inference on the edge exploiting TinyML capabilities for UAVs. Drones, vol. 5, no. 4, p. 127, 2021. 19. G. C. Goodwin, S. F. Graebe and M. E. Salgado. Control system design. IEEE Control Systems, United States, 2007. 20. I. C. Pérez, D. Flores-Araiza, J. A. Fortoul-Díaz, R. Maximo and H. G. Gonzalez-Hernandez. Identification and PID control for a quadrocopter. In: 2014 International Conference on Electronics, Communications and Computers (CONIELECOMP). IEEE, United States. 2014. 21. W. Bolton. Programmable logic controllers fourth edition. In: Africa’s Potential for the Ecological Intensification of Agriculture, Elsevier, Amsterdam, 2006. 22. D. Nixon, A. Mironov, M. A. M. Martinez, J. Blackman, Codecae, B. Luckcuck, A. Christensen, D. Clifton, M. Keller, R. Coeffic and S. Windoffe. BetaFlight. Available from: https://github.com/ betaflight/betaflight [Last accessed on 2022 Dec 20]. Table 3: The output of PID for the drone movements Movement P I D Roll 58 124 31 Pitch 63 131 33 Yaw 58 124 0