Microsoft Word - ETASR_V13_N3_pp10870-10874 Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10870-10874 10870 www.etasr.com Almukadi: Smart Scarf: An IOT-based Solution for Emotion Recognition Smart Scarf: An IOT-based Solution for Emotion Recognition Wafa Almukadi Department of Software Engineering, College of Computer Sciences and Engineering, University of Jeddah, Saudi Arabia wsalmukadi@uj.edu.sa (corresponding author) Received: 12 April 2023 | Revised: 24 April 2023 | Accepted: 26 April 2023 Licensed under a CC-BY 4.0 license | Copyright (c) by the authors | DOI: https://doi.org/10.48084/etasr.5952 ABSTRACT This paper presents the design rationale of a color-changing scarf depending on the emotional state it detects. The goal of the design is to associate people to show up their emotional state especially those who have difficulties in expressing their feelings, such as elderly people, children, people with special needs, and people who have to wear a face mask, which covers their facial expression, as it is common during the COVID-19 pandemic period. Wearing a smart emotional scarf provides a context of the individual's emotional state that might help surrounding people in dealing with them. Our design uses a heart rate sensor and a skin sensor to detect and recognize emotional information. The scarf will change its color based on the emotional state it detects (neutral, angry, happy, and sad). An interface allows the provision of emotional data and displays emotional statistics and daily history. We went through a user-centered design process adopting wearable technology guidelines. A proposed prototype of the smart scarf is presented called Scarfy. A user evaluation is conducted showing that wearing the scarf with changing colors based on emotional status is a good solution to express feelings with comfort and wearing it might enhance social engagement. Keywords-wearable technology; design process; emotion recognition; Internet of Things (IOT) I. INTRODUCTION Wearable technology has evolved along with wearable sensors and textile technologies, and they are now a part of our daily life. In the subject of human-computer interaction, using wearable technology to provide wearers with emotional support has been seen as an essential research question in Human- Computer Interaction (HCI). In addition, to improve one's own emotional health, it is necessary to have an impact on others' emotional state through interactions in a social setting. Moreover, in the field of ambient assisted living, people with Mild Cognitive Impairments (MCI), Alzheimer's disease, or dementia, require supervision in their everyday activities. The need to develop an innovative approach in order to explore ways of working with elder people and their formal and informal caregivers, is emphasized [1]. Providing information on both physical and emotional statuses positively affects multiple stakeholders and facilitates communication. Today technologies facilitate monitoring and analyzing body data to recognize a person’s feelings and moods. Interaction designers face new challenges when designing wearable technology. Beyond those of self-emotional awareness and management, other challenges and principles are raised by the design of wearable technology. Designers need to adopt a user-centered methodology to find the important touch points with users that help in finding recommendations for tech designers. Smart wearable devices are becoming an increasingly important part of the Internet of Things (IoT) technology. Smart watches, wrist bands, headsets, and earbuds are some of the wearable technologies developed for different applications. Authors in [2] classify the smart wearable devices into major clusters based on their applications: Health, sports, daily activity, tracking and localization, and safety. Moreover, IoT-based wearable technology can be improved further to save human lives by detecting in advance health risk conditions such as heat strokes [3].  The major challenges of wearable IoT devices are data resolution of wearable sensors, power consumption, wearability, safety, security, regulation, and privacy [4]. In this paper, an emotion smart scarf called Scarfy is presented. Scarfy is connected to an LED light that can change its color based on the wearer's feelings. The scarf analyzes the wearers' feelings through their vital signs and tries to help them improve their emotional state and facilitate the elucidation of their feelings to others. In addition, the scarf is connected to a mobile application via Bluetooth to store the user's current emotional status. The application helps determines the users’ emotional status and tries to predict different emotional statuses over a period. These data can be used in future work to determine user behavior and predict a user's emotional status. We add a new classification to wearable application, "human emotions". During the design process, focus was to both challenges, data resolution of the wearable sensors and wearability.  Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10870-10874 10871 www.etasr.com Almukadi: Smart Scarf: An IOT-based Solution for Emotion Recognition II. RELATED WORK A. Emotion and Technology Heart Rate Variability (HRV) has been used to quantify controlled emotional reactivity [5]. HRV tracks the ebbs and flows of heart rate. A rise in HRV is associated with positive stimuli, while a drop in HRV is associated with negative stimuli. Hence, HRV can be used to gauge a person's emotional state. Authors in [6] looked for ways to create a wearable display that changes colors quickly. Their AmbiKraf non- emissive analog fabric display was developed using thermochromic inks and incorporated semiconductor peltier junctions. The controller enables the thermochromic ink to display several colors while regulating the temperature in the soft cloth [7]. Authors in [2] studied the association between heart rate changes and emotions changes. They considered three emotions: sad, happy, and neutral. The study uses HRV extracted from the PPG signal and analyzes its relationship with feelings. Ten characteristics were computed and analyzed in each measurement cycle. The project is similar to the work in this paper by studying the relationship between heart rate changes and emotional changes using electrocardiograms (ECGs). But our project Scarfy uses a skin sensor as well and produces a response based on the results. Scarfy is also a wearable smart device that can be used by anyone. In our project we will use the same approach to detect emotions based on HRV. B. Emotion and Therapy throught a Smart-scarf Authors in [5] looked for ways to affect a user's mood using bodily data. Twelve people aged from 20s to 40s were chosen to discuss their emotional experiences. All participants were college students and employees. Numerous concepts were discussed before presenting them to the same group of people: a clever hair band, a bracelet, a helmet, a vest, and a scarf. Most of the participants opted to wear a scarf since it is less noticeable and could be worn daily. Researchers suggested to create a smart-scarf prototype called Smiley to help individuals comprehend self- and emotion-consciousness, improve communication, and strengthen social relationships by altering its color. The interaction concept was developed into three scenarios:  Self-scenario  One-to-one scenario  One-to-many scenario A wearable device must be flexible, lightweight, pleasing, conductive, and fairly priced. Its design considers connecting to a heart rate sensor module, a skin conductance sensor module, a color sensor module, and an IR transceiver to a LilyPad Arduino. In our project, we will currently focus on the self- scenario. C. Emotion and Color Combination A color-combination emotion model was created in [8]. It was made up of a three-dimensional axis that depicts the parameters soft-hard, light-heavy, and splendid-sober. The color combinations correspond to nine different mood categories. In his work A Book of Colors, Shigenobu Kobayashi examined numerous color combinations while making allusions to various emotion adjectives [9]. The primary focus is given on the interior design, which motivated us to investigate how color combinations and emotions relate to one another. Table I summarizes the similarities and differences between the related work and our project. TABLE I. RELATED WORK SUMMARY Related work Difference Similarities Detect emotions changes using heart rate changes [2] The project depends only on a heart rate sensor and the result is shown in a screen. Our project uses both heart rate and skin sensors then produces a response based on the results on a smart wearable device. Smilar to our project in the motivation of studying the relationship between heart rate and emotions changes using ECG. Emotion therapy through a smart- scarf [5] The interaction concept is developed into 3 scenarios: Self, One-to-one, and One- to-many. Our project will develop the Self scenario. Create the same wearable device connecting to a heart rate sensor and a skin sensor. User experience on a smart scarf [8] Focus is given on enhancing communication between the scarf wearer and the surrounding parties, while our project does not consider the surrounding parties at this stage. A smart wearable is created that will assist people become more conscious of their emotions in a group surroundings while also enhancing good emotions and reducing negative ones III. DESIGN PROCESS Design wearable technology is beyond combining electronic components with fashionable clothing, however it should concern physical form, electronic components, Graphical User Interface (GUI), physical interface, and embodied product value [10]. Furthermore, the design of wearable technology needs to consider the collection of bodily data and their processing, considering some design principles such as comfort, contextual-awareness, affordance customization, ease of use, pleasant, ergonomy, wearability, and fashion [11]. A. User-Centered Design Process We investigated the use of body data to influence a user’s mood and allowing the surrounding environment to be aware of one’s emotional status. With a web survey method, we reached 269 participants to answer a survey of 14 questions and share their emotional experience. All participants were college students and staff members, and their ages ranged from 17 to 45. They were questioned about what their bodies had gone through when being in depressing or exhilarating situations and how do colors affect their emotional experience. Collecting data helped us determine the project requirements and specifications. B. Biometrics Data Gathering and Analysis The heart, through contraction and diastole, pumps blood to all parts of the body. When blood is pumped into the body, it produces a pulse in the arteries. This pulse represents the Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10870-10874 10872 www.etasr.com Almukadi: Smart Scarf: An IOT-based Solution for Emotion Recognition number of heartbeats. The healthy heart rate for resting adults is 60-80 beats per minute (BPM) [12]. Changes in heart rate can be seen with a typical ECG. There are factors that affect the resting heart rate, including heart problems and emotional outbursts. In our project, we focus on the relationship between changes in the heart rate and the emotional state. McCraty [13] observed a clear change in the pattern of the heart's rhythm due to the change in feelings. It was found that negative emotions such as anxiety and anger cause irregular heartbeats, while positive emotions such as sympathy and love produce high heartbeat rhythms. So, we chose the heart rate as one of the indicators to differentiate between emotions in our project. In addition, in terms of identifying human sensations, skin temperature is an effective indicator. The association between stressful work and skin temperature was investigated and it was discovered that when people are stressed, tense, or have other unpleasant feelings, their skin temperature rises [12]. For accurate results of the wearer's emotional state, we considered combining both sensors, heart rate and skin temperature. They can be used to classify the emotional state in 4 categories (neutral, happy, sad, and angry) [14] as shown in Table II. TABLE II. EMOTION STATES ACCORDING TO THE HEART RATE AND SKIN TEMPERATURE State BPM Skin temperature Natural 50 - 80 36.5 - 37.5 °C Happy 90 - 100 37.5 - 38 °C Sad 80 - 90 35 - 36 °C Angry 100 - 120 37 - 38.5 °C Fig. 1. Scarfy changes color base on emotional state. Our project will adopt the techniques developed in [14]. We will represent the emotional state with different colors as following: white for neutral, red for angry, blue for happy, and yellow for sad (Figure 1). C. Interface Design We chose cotton fabric to design our smart scarf Scarfy (Figure 2). The main components of the system are: 1. A scarf made of cotton fabric. 2. Heart rate MAX30100 Pulse Oximeter sensor: is an I2C- based low-power plug-and-play biometric sensor. 3. Skin temperature sensor: an infrared thermometer for non- contact temperature measurements HiLetgo GY-906 MLX90614ESF non-contact infrared. 4. LED light strip: WS2812B. 5. Bluetooth: HC-05. 6. Arduino: An open-source single-board microcontroller appropriate for low budget, small electronics projects. HRV and skin temperature sensors are used to detect scarf wearer's emotions. They are comfortable, inexpensive, and can be easily attached to the scarf. The Arduino board is used to read the input data from sensors and control them. The small size, compact design, and advanced features of the Arduino make it an appropriate choice for wearable technology [15]. The system architecture demonstrates the position of the two sensors (heart rate and skin temperature). The sensors will sense heart rate and skin temperature and send the data to the Arduino, which will collect and analyze them to determine the emotional state. After that, the LED color will change based on the current emotional state of the wearer and send the state to the "Scarfy" application via Bluetooth. The application will send the state to Firebase to store the user status and retrieve when needed for open daily/weekly analysis (Figure 3). Fig. 2. System architecture and design for Scarfy. IV. WORKING PROCEDURE In order to detect the emotional state and change led color, we adopt the methodology of [14]. The following algorithm describes the working procedure: If the heart sensor between (100 and 120) and skin temperature between 37 - 38.5 °C Then emotional state is "angry" and led = red Else if heart sensor between (91 and 100) and skin temperature between 35 - 36 °C Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10870-10874 10873 www.etasr.com Almukadi: Smart Scarf: An IOT-based Solution for Emotion Recognition Then emotional state is "sad" and led = yellow Else if heart sensor between (80 and 90) and skin temperature between 37.5 - 38 °C Then emotional state is "happy" and led = green Else if heart sensor between (50 and 80) and skin temperature between 36.5 – 37.5 °C Then emotional state is "neutral" and led = white Fig. 3. System architecture connecting to mobile application. V. USER EVALUATION Scarfy was tested on 10 participants aged between 20 and 25. The participants were seated on an empty lab except for the observer and were asked to wear Scarfy. First, each user was required to create an account at the Scarfy application by entering his email and password and filling up the profile page. After that, the user has to select some preferred activities regarding his emotional status such as: "What activities help you relax? Eat an ice cream, watch a movie, read a book, play video games, go on a picnic, and talk to friends." The user’s preferred activities were attached next to user emotional status as a suggestion message to enhance user’s negative mood and maintain good mood (Figure 4). Subsequently, different movie clips were played with different contents. The contents were intended to affect the emotions of participants. When the participant was watching the clip and Scarfy detected changes in the emotional status, the user was prompted to answer an online questionnaire to evaluate the prototype and provide feedback. Figure 5 shows two participants wearing Scarfy while being in different emotional statuses during watching movie clips with different contents. Moreover, the mobile application provides a feature to users that allows them to view their emotional states during the day or the week (Figure 6). This feature allows the users to track their emotional states and be aware of which activities affected their emotions negatively or positively. This analysis may be used as raw data for future work. Fig. 4. User interface of the sign up page of the Scarfy mobile application. Fig. 5. Participants during the user evaluation session. VI. DISSCUSSION AND CONCLUSION This paper focuses on the use of wearable technology to provide wearers with emotional support which is an essential research question in the HCI field. Our evaluation indicates that users were pleased with the smart scarf’s appearance, comfort, and engagement. The smart scarf was proven simple to use while it assists people to track and control their emotions. At the same time, it provides a good channel for people to show their feelings to the surrounding people, thus facilitating communication. However, inaccurate readings may Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10870-10874 10874 www.etasr.com Almukadi: Smart Scarf: An IOT-based Solution for Emotion Recognition result from the limits of skin conductance and heart rate sensors. Moreover, not all body emotions could be distinguished by the two sensors alone. Fig. 6. Daily (left) and weekly (right) emotional state analysis. For future work, we may consider employing a variety of channels, such as breathing and blood pressure pulse, to detect emotions more emotions with more accuracy. Moreover, we should address the gender and culture differences in color and privacy. Furthermore, we would like to consider the one to one and group scenarios where users who wear Scarfy can communicate together via their scarves sharing their emotional states and allow sending messages based on their emotional status. In general, Scarfy aims to enhance social bonds and facilitate interaction. The stored emotional data for users may be an opportunity for further artificial intelligent studies. ACKNOWLEDGMENT The author would like to thank Amal Almaabady, Bayan bin Tahin, Ferdous Babttat, and Manar Alsefri for their contribution in implementing the prototype of Scarfy during their senior project. REFERENCES [1] D. Schnelle-Walka et al., "Proceedings of the 1st Joint Workshop on Smart Connected and Wearable Things 2016," IOWIT, 2016, https://doi.org/10.1145/2876456.2882849. [2] R. Rakshit, V. R. Reddy, and P. Deshpande, "Emotion detection and recognition using HRV features derived from photoplethysmogram signals," in Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems, New York, NY, USA, Aug. 2016, Art. no. 2, https://doi.org/10.1145/3009960.3009962. [3] F. John Dian, R. Vahidnia, and A. Rahmati, "Wearables and the Internet of Things (IoT), Applications, Opportunities, and Challenges: A Survey," IEEE Access, vol. 8, pp. 69200–69211, 2020, https://doi.org/ 10.1109/ACCESS.2020.2986329. [4] S. Javed, S. Ghazala, and U. Faseeha, "Perspectives of Heat Stroke Shield: An IoT based Solution for the Detection and Preliminary Treatment of Heat Stroke," Engineering, Technology & Applied Science Research, vol. 10, no. 2, pp. 5576–5580, Apr. 2020, https://doi.org/ 10.48084/etasr.3274. [5] B. M. Appelhans and L. J. Luecken, "Heart Rate Variability as an Index of Regulated Emotional Responding," Review of General Psychology, vol. 10, no. 3, pp. 229–240, Sep. 2006, https://doi.org/10.1037/1089- 2680.10.3.229. [6] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, "Human Emotion Recognition: Review of Sensors and Methods," Sensors, vol. 20, no. 3, Jan. 2020, Art. no. 592, https://doi.org/10.3390/s20030592. [7] R. L. Peiris, M. J. Tharakan, N. Fernando, and A. D. Chrok, "AmbiKraf: A Nonemissive Fabric Display for Fast Changing Textile Animation," in 2011 IFIP 9th International Conference on Embedded and Ubiquitous Computing, Melbourne, VIC, Australia, Jul. 2011, pp. 221–228, https://doi.org/10.1109/EUC.2011.13. [8] Y.-J. Lee and J. Lee, "The development of an emotion model based on colour combinations," International Journal of Consumer Studies, vol. 30, no. 2, pp. 122–136, 2006, https://doi.org/10.1111/j.1470-6431. 2005.00457.x. [9] S. Kobayashi, A book of colors: Matching colors, combining colors, color designing, color decorating. Kodansha, 1987. [10] J. Maeda, The Laws of Simplicity. Cambridge, MA, USA: MIT Press, 2020. [11] V. G. Motti and K. Caine, "Human Factors Considerations in the Design of Wearable Devices," Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 58, no. 1, pp. 1820–1824, Sep. 2014, https://doi.org/10.1177/1541931214581381. [12] M. T. Quazi, "Human emotion recognition using smart sensors," M.S. thesis, Massey University, Palmerston North, New Zealand, 2012. [13] S. Xefteris, N. Doulamis, V. Andronikou, T. Varvarigou, and G. Cambourakis, "Behavioral Biometrics in Assisted Living: A Methodology for Emotion Recognition," Engineering, Technology & Applied Science Research, vol. 6, no. 4, pp. 1035–1044, Aug. 2016, https://doi.org/10.48084/etasr.634. [14] G. Kaur and P. S. Pandey, "Emotion Recognition System using IOT and Machine Learning - A Healthcare Application," in Proceedings of the 23rd Conference of Open Innovations Association FRUCT, Helsinki, Uusimaa, FIN, Aug. 2018, pp. 465–470. [15] S. Zafar, G. Miraj, R. Baloch, D. Murtaza, and K. Arshad, "An IoT Based Real-Time Environmental Monitoring System Using Arduino and Cloud Service," Engineering, Technology & Applied Science Research, vol. 8, no. 4, pp. 3238–3242, Aug. 2018, https://doi.org/10.48084/ etasr.2144.