DOI: 10.13102/sociobiology.v67i4.5860Sociobiology 67(4): 566-571 (December, 2020) Open access journal: http://periodicos.uefs.br/ojs/index.php/sociobiology ISSN: 0361-6525 Introduction The colony sizes of managed Apis mellifera have been declining worldwide (Potts et al. 2010). Researchers are developing monitoring systems based on beehive sounds for devising the possible strategies for the management and conservation of the dynamic pollinators (Meikle & Holst, 2015; Mezquida & Martínez, 2009; Sharif et al., 2020). A USDA Small Business Innovation Research (SBIR) project focused on the recognition of the acute as well as chronic exposure of honeybees to pesticides, including neonicotinoid pesticides. Preliminary outcomes reveal that bee colonies tend to become noisier upon exposure to organic pesticides, while silent upon exposure to neonicotinoid pesticides (Seccomb, 2015). The use of beehive sound recordings has been developed for the detection of chemical-laced airborne threats under SBIR agreement from the United States Army Center for Abstract As the study of honey bee health has gained attention in the biology community, researchers have looked for new, non-invasive methods to monitor the health status of the colony. Since the beehive sound alters when the colony is exposed to stressors, analysis of the acoustic response of the colony has been used as a method to identify the type of stressor, whether it is chemical, pest, or disease. So far, two feature sets have been successfully used for this kind of analysis, being these low-level signal features and Mel Frequency Cepstral Coefficients (MFCC). Here we propose using soundscape indices, developed initially to delineate acoustic diversity in ecosystems, as an alternative to now used features. In our study, we examine the beehive acoustic response to trichloromethane laced-air and blank air and compare the performance of all three feature sets to discern the colony’s sound between the hive being exposed to the chemical and not. Our results show that sound indices overperform the alternative features sets on this task. Based on these findings, we consider sound indices to be a valid set of features for beehive sound analysis and present our results to call the attention of the community on this fact. Sociobiology An international journal on social insects MZ Sharif1, 2, F Wario3, N Di1,2, R Xue1,2, F Liu1,2 Article History Edited by Denise Alves, USP, Brazil Received 22 September 2020 Initial acceptance 20 November 2020 Final acceptance 07 December 2020 Publication date 28 December 2020 Keywords Apis mellifera, bioacoustics, low-level signal features, machine learning, trichloromethane, soundscape index Corresponding author Fanglin Liu https://orcid.org/0000-0002-8371-6316 Institute of Technical Biology and Agriculture Engineering Hefei Institutes of Physical Sciences Chinese Academy of Sciences Hefei, Anhui, P.R. China. E-Mail: flliu@ipp.ac.cn Environmental Health Research. The most dynamic thing about the results obtained from the investigates is that the bee colony sound not only alters in response of exposure to variant stressor, but in some cases also alters to delineate the particular type of chemical, pest, and disease (Bromenshenk et al., 2009, 2004). A beehive is a small ecosystem, consisted of 30,000-50,000 individual bees. The soundscape indices, used to describe the sound scene, may be suitable to delineate the beehive sounds emitted by different individuals. But their efficacy in classifying beehive audio samples has not been explored so far. Audio-based classification models have been explored to detect the statuses of a beehive, such as queenless (Howard et al., 2013), swarming (Ferrari et al., 2008), and diseases or pests (Robles-Guerrero et al., 2017) etc. However, the automatic classification of beehive audio samples remains a challenge yet. 1 - Institute of Technical Biology & Agriculture Engineering, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, P.R. China 2 - University of Science and Technology of China, Hefei, 230026, P.R. China 3 - University of Guadalajara Av. Juárez N° 976, Col. Centro, 44100, Guadalajara, Jalisco, México RESEARCH ARTICLE - BEES Soundscape Indices: New Features for Classifying Beehive Audio Samples Sociobiology 67(4): 566-571 (December, 2020) 567 Classification of audio samples involves two stages, the first in which certain features are extracted from waveform files, and the second in which a classification model is built on the extracted features. Several classification schemes have been proposed, including statistic models, such as Linear Discrimination Analysis (Qandour et al., 2014) and Self-Organizing Maps (Howard et al., 2013), and Machine Learning Algorithms (Nolasco & Benetos, 2018), such as Support Vector Machine (Qandour et al., 2014), Random Forest (Robles-Guerrero et al., 2017) and Neural Network (Rybin et al., 2017). It has been shown that the schemes are not different in the classification accuracy (Qandour et al., 2014). Given the current state of audio classifiers, further advances could be made by developing powerful features, rather than building new classification schemes. So far, two feature sets are used for the classification of beehive audios. They are the low-level signal feature set, including Zero-Crossing Rate, Bandwidth, Spectral Centroid, and Signal Energy (Qandour et al., 2014), and the Mel spectra set, which are dozens of Mel Frequency Cepstral Coefficients (MFCC) (Robles-Guerrero et al., 2017) (Nolasco et al., 2019). Recently, ecologists have been developing soundscape indices to assess biodiversity at landscape scale (Mammides et al., 2017). In the present study, we examine the beehive sound response to an organic chemical compound (trichloromethane [CHCl3]), as the beehive sonograms are altered when bees are exposed to different chemicals (Bromenshenk et al., 2015). We also compare the performance of the soundscape indices with that of the low-level signal properties in discriminating beehive sound response to this chemical. The application of soundscape indices in classifying beehive audio data is also discussed. Material and methods Experimental Setup The experiment was conducted in the Spring and the Summer, 2019, respectively, at the Hefei Institutes of Physical Sciences, Chinese Academy of Sciences. A beehive with A. mellifera was kept in a lab room. A pre-cut circular hole in the glass window of the room fits tightly around a transfer entrance/exit tube. The bees were allowed to come and go through the transfer tube freely. An outdoor-facing landing platform was fixed outside right below the entrance/exit end of the transfer tube, giving our bees a convenient perch for takeoff and landing. The beehive was set for one week. A vacuum pump was used to pump the air at a ~ 5 meters distance. The air was first pumped to a glass jar via a flexible plastic pipe in which some filters with or without trichloromethane were placed. Later, the air was pumped to the beehive via beehive entrance. The content of trichloromethane in the glass jar was 250 mg/ m3, which is slightly higher than the maximum allowable concentration in the open air (240 mg/m3). An airflow rate of 0.4 m/s was controlled by an Air-Flow Meter. Data Collection Audio data were collected at night time in good weather days with a microphone, positioned inside the hive to avoid propolization, at a sampling rate of 22.05 kHz (16 bits, one channel). To ensure the data are correctly labeled with ventilating trichloromethane-laced air, only the continuous recordings within 3-23 min after the air was released to the hive were used. The audio files were saved in a waveform (wav). The hive air was expelled out by a fan at the hive entrance for two days before blank air was pumped. The procedure alternated ventilating trichloromethane-laced air, and ventilating blank air was repeated five times (3 times for the Spring and two times for the Summer). Feature Extraction Each 20-min audio file, either labeling with blank or trichloromethane-laced air, was divided into 20-s samples without overlap. The common low-level signal features, such as Zero Crossing Rate (ZCR) and Spectral Centroid (SC), and 12 Mel-Frequency Cepstral Coefficients, were extracted from each 20-s sample with a 20-ms frame with 50% overlap. Mel-frequency cepstral is a standard extraction approach for similar classification tasks (Robles-Guerrero et al., 2017). In the R programming language (TeamRCore, 2015), such features were extracted according to previously described methodology (Li et al., 2001). The common soundscape indices, Acoustic Complexity Index (ACI), Acoustic Diversity Index (ADI), Acoustic Evenness Index (AEI), and Bioacoustic index (BI), were calculated with the “soundecology” package (Villanueva-Rivera and Pijanowski 2016). ACI was analyzed with a cluster size of 10 seconds and limited to 6000 Hz. The maximum frequency of ADI and AEI is set to 6000 Hz. The size of the frequency bands is set to 1000 Hz. Other indices were calculated with the default settings. Feature Performance Feature performance was estimated using random forests. Briefly, 75% of the 20- audio samples were randomly chosen as a training group, the others as a test group. For the training group, the features of an audio sample were grouped into either blank ventilating air or trichloromethane-laced air. A binary classification model was built based on the features of the audio samples by the randomForest (RF) package “v4.6-12” (Liaw & Wiener, 2002). Details about this classification method are described in a publication (Breiman, 2001). The trained RF model classified each audio sample in the test group to find which class it most probably belongs to. The overall misclassification rate, estimated from the confusion matrix, was used as an essential performance measure. Random forest possesses a couple of methods to evaluate the importance of every predictor variable in the model; the methods included are Mean Decrease in Accuracy (MDA) and Mean Decrease in Gini (MDG) (Cutler et al., 2007; Calle & Urrea, 2011). But, we had chosen MDA in our current investigation to predict the importance of indices (Fig 1A, D & G) as computed by random forest. MZ Sharif, F Wario, N Di, R Xue – Monitoring honeybee colony: the role of acoustic features568 Fig 1. The importance order of three feature sets and their changes with time. Black circles denotes ventilating blank air, red circles denotes trichloromethane-laced air. Each point represents the datum extracted from a 20-s audio sample. CZR: Zero- Crossing Rate, CS: Spectral Centroid, C[1] to C[12]: MFCC Coefficients, sp.ent: Spectral Entropy, meanfreq: mean frequency, meanAmp: mean amplitude, Energy: energy of singal, RMS: root-mean-square level, ACI: Acoustic Complexity Index, ADI: Acoustic Diversity Index, AEI: Acoustic Evenness Index, BI: Bioacoustic Index Results Concerning time, each treatment (i.e., blank air and trichloromethane laced-air) yielded significant differences among all the indices. Regarding the upper panels, they showed the importance order of the low-level signal features and their changes with time (Fig 1B and C). ZCR and SC were the most critical features (Fig 1). They significantly fluctuated with time (Fig 1B and C); either blank air or trichloromethane-laced air was ventilated into the beehive. Visual examination designates that the values of ZCR and SC for trichloromethane laced-air fluctuate and decrease with time and remain below their values for blank air. Based on the two features, the beehive sound responses to the two air types could not easily be discriminated against via a classification model, such as linear discrimination analysis. The middle panels showed the importance order of MFCCs and their changes with time. The MFCC coefficients (C[1] and C[3]) were the most fundamental features (Fig 1D). The graphic representation exhibits that the values of C[1] and C[3] features for chemical laced-air fluctuated, but the former upsurge and later declined with time. Inversely, for blank air, the C[1] feature oscillated below the chemical laced-air, while the values of C[3] oscillated above the chemical laced-air. The changes detected in C[1] coefficient for trichloromethane and blank air were clear to observe the acoustic variations, but the changes seen in C[3] were mixed at some points (Fig 1E and F). According to the classification accuracy, MFCC showed good performance than the low-level features, but the low performance was noted compared to the soundscape indices (Table 1). Anyhow, the mean decrease in accuracy values for C1 and C3 were higher than other coefficients, which represent Sociobiology 67(4): 566-571 (December, 2020) 569 them as a critical feature for acoustic discrimination (Fig 1D). Based on MFCC features, especially C[1], the beehive sound responses to the two air types can be discriminated. Regardless of upper and middle panels, the lower panels showed more importance in soundscape indices and their change time. ACI and ADI were also the most significant indices (Fig 1H and I); based on these indices, beehive sound responses to blank air remained stable. However, bees response to trichloromethane-laced air dramatically fluctuated, either below (Fig 1I for ACI) or above (Fig 1H for ADI), the response of bees to blank air. By visual examination, the soundscape indices were more discriminative power than the low-level signal features and MFCCs for binary classification. The classification accuracy calculated using RF model is summarized in Table 1. In terms of classification accuracy, the soundscape indices have 9.99% to 11.66% better performance than other feature sets including MFCC, and Low-level signal features, respectively. Based on the four crucial low-level signal features (ZCR, SC, mean free, and sp.ent) extracted from each 20-s audio sample, the classification accuracies were 75% for trichloromethane-laced air and 81.67% for blank air, respectively. Based on the four important MFCCs, the classification accuracies were 76.90 % for trichloromethane- laced air and 80.00% for blank air, respectively. According to the soundscape indices (ACI, ADI, AE, BI), the accuracies were 88.46 % for trichloromethane-laced air and 91.67 % for blank air, respectively. Furthermore, the mean decrease in accuracy values for ACI and ADI is higher than other feature sets like BI, AEI, SC, and others (Fig 1A, D & G). That is why both the soundscape indices (ACI and ADI) were designated as the most central indices, followed by other indices and feature sets. Discussion Audio-based beehive monitoring becomes a partial solution to precision apiculture (Ferrari et al., 2008). Several acoustic features have been explored to classify beehive audio samples (Cecchi et al., 2019; Qandour et al., 2014). In this study, we show that the soundscape indices are more potent than the low-level signal features and MFCCs in discriminating bee response to chemicals. Anyhow, MFCC features also achieved good classification accuracy than the low-level features, although less than the soundscape indices (Fig 1E and 1F). Similar results for classification accuracy of MFCC features were presented in a research by Zgank (2019). This research describes that with the medium and high complexity acoustic models, the MFCC features achieved 75.43% and 82.27% classification accuracy without cepstral mean normalization, and 74.81% and 81.96% with it, respectively. But, the soundscape indices may be more effective and can be used as new acoustical features to classify the beehive audio data. Why are soundscape indices suitable for the classification of beehive sounds? Temporal variations of the basic features are essential for classification (Breebaart & McKinney, 2004). The greater the variability of the basic features, the more likely it achieves a high classification accuracy. Unlike low- level signal features, which are extracted from the stationary periods (Robles-Guerrero et al., 2017), the soundscape indices are designed to assess changes in acoustic energy in time or frequency domain (or both) as an analog for the number of vocalizing animals in an area (Parsons et al., 2016). The extraction process for the soundscape indices includes signal variations as much as possible. Therefore, the soundscape indices are suitable representations in the complex context of a beehive sound scene. Most studies on avian soundscapes use 1-min audio file to extract soundscape indices. In our research, we found that the soundscape had a time scale. For the Spring samples, as the frames increased from 20 to 30 s, the classification accuracy for trichloromethane air remained 88.46% (Table 1). This is why we chose 20 s as the frame size for the calculation of the soundscape index. Whether it is the optimum sample size for calculation of the soundscape indices should be further examined. Nonetheless, from a perceptual point of view, this is a remarkably short section of audio files on which the soundscape indices are calculated. Improvements in classification performance could indeed be made if classification were based on more than a one-time scale. Table 1. Comparison of the discriminative power between three feature sets based on Random Forest. Low-level signal features: spectral centroid (SC), zero-crossing rate (ZCR), mean frequency (meanfreq), and spectral entropy (sp.ent); mel frequency cepstral coefficient (MFCCs): C[1], C[3], C[7] and C[5]; Soundscape indices: Acoustic Complexity Index (ACI), Acoustic Diversity Index (ADI), Acoustic Evenness Index (AEI), and Bioacoustic index (BI). Feature set Sample size Frame size Classification accuracy Trichloromethane-laced air Blank air Low-level signal features 20 s 20 ms 75.00% 81.67% MFCC 20 s 20 ms 76.90% 80.00% Soundscape indices 20 s - 88.46% 91.66% Soundscape index 30 s - 88.46% 86.67% MZ Sharif, F Wario, N Di, R Xue – Monitoring honeybee colony: the role of acoustic features570 In short, we show that the performance of the soundscape index is superior to that of the standard low-level signal features in classifying the beehive audio samples. The soundscape indices, which include signal variations as much as possible, may be necessary for their suitability to represent the beehive-inside complex sound scene. Several chemical pesticides (chlorpyrifos, cypermethrin, carbofuran, bifenthrin, clothianidin, and others) have previously been checked for harmful effects on the lives of honey bees (Yao et al., 2018; Yang et al., 2020; Dai et al., 2010). Therefore, we recommend that researchers should use soundscape features as a valuable tool for evaluating the health of colonies in future studies. Moreover, the automatic classification of beehive audio samples can be optimized using the soundscape features. Acknowledgments This work was supported by the Key Research Program of the Chinese Academy of Sciences (Grant no. KGFZD-135-17-010). The authors would like to thank Zhuqing Lv, Sabah Mushtaq Puswal, Jianjun Liao, and Fenmei Wang for their assistance in the trails. Authors' contribution FL and MZS conceived and designed the analysis, and did the data collection. FL and MZS wrote the manuscript. MZS refined the manuscript with grammatical corrections and improved the description of the result section of the manuscript. FL did the data analysis. FW helped in manuscript preparation, write up and suggesting improvements in data analysis. ND and RX revised the manuscript and contributed to significant improvements. All the authors coordinated to revise and approve the final version of the manuscript. References Breebaart, J. & McKinney, M. (2004). Features for audio classification. In Verhaegh W, Aarts E, Korst J (eds) Algorithms in Ambient Intelligence (pp. 113-129). Netherland: Kluwer Academic Publishers. Breiman, L. (2001). Random Forests. Machine Learning 45: 5-32. Bromenshenk, J., Henderson, C., Seccomb, R., Welch, P., Debnam, S. & Firth, D. (2015). Bees as biosensors: chemosensory ability, honey bee monitoring systems, and emergent sensor technologies derived from the pollinator syndrome. Biosensors, 5: 678-711. doi: 10.3390/bios5040678 Bromenshenk, J.J., Henderson, C.B., Seccomb, R.A., Rice, S.D. & Etter, R.T. (2009). Honey bee acoustic recording and analysis system for monitoring hive health. U.S. Patent 7,549,907 B2 Bromenshenk, J.J., Seccomb, R.A., Rice, S.D., & Etter, R.T. (2005). Honey bee monitoring system for monitoring bee colonies in a hive. U.S. Patent 6,910,941 B2. Calle, ML. & Urrea, V. (2011). Letter to the editor: Stability of Random Forest importance measures. Briefings in Bioinformatics, 12: 86-9. doi: 10.1093/bib/bbq011 Cecchi, S., Terenzi, A., Orcioni, S. & Piazza, F. (2019). Analysis of the sound emitted by honey bees in a beehive. In Audio Engineering Society Convention (p. 147). Cutler, D.R., Edwards, T.C, Beard, K.H, Cutler, A., Hess, K.T., Gibson, J. & Lawler, J.J. (2007). Random forests for classification in ecology. Ecology, 88: 2783-2792. doi: 10. 1890/07-0539.1 Dai, P.L., Qiang, W. & Hu-Sun, J. (2010). Effects of sublethal concentrations of bifenthrin and deltamethrin in fecundity, growth, and development of the honeybee Apis mellifera ligustica. Environmental Toxicology and Chemistry, 29: 644- 649. doi: 10.1002/etc.67 Ferrari, S., Silva, M., Guarino, & M., Berckmans, D. (2008). Monitoring of swarming sounds in bee hives for early detection of the swarming period. Computer in Electronics and Agriculture, 64: 72-77. doi: 10.1016/j.compag.2008.05.010 Howard. D., Duran, O., Hunter, G. & Stebel, K. (2013). Signal processing the acoustics of honeybees (Apis mellifera) to identify the ‘Queenless’ state in hives. Proceedings of the Institute of Acoustics, 35: 290-297. Li, D., Sethi, I.K., Dimitrova, N. & McGee, T. (2001). Classification of general audio data for content-based retrieval. Pattern Recognition Letters 22: 533-544. doi: 10.1016/S0167- 8655(00)00119-7 Liaw, A., & Wiener, M. (2002). Classification and regression by random forest. R News, 2: 18-22. Mammides, C., Goodale, E., Dayananda, S.K., Kang, L. & Chen, J. (2017). Do acoustic indices correlate with bird diversity? Insights from two biodiverse regions in Yunnan Province, south China. Ecological Indicators, 82: 470-477. doi: 10.1016/j.ecolind.2017.07.017 Meikle, W. & Holst, N. (2015). Application of continuous monitoring of honeybee colonies. Apidologie, 46: 10-22. doi: 10.1007/s13592-014-0298-x Mezquida, D.A. & Martínez, J.L. (2009). Platform for beehives monitoring based on sound analysis. A perpetual warehouse for swarm’s daily activity. Spanish Journal of Agriculture Research, 7: 824-828. Nolasco, I. & Benetos, E. (2018). To bee or not to bee: Investigating machine learning approaches for beehive sound recognition. arXiv 1811.06016. Nolasco, I., Terenzi, A., Cecchi, S., Orcioni, S., Bear, H.L., & Benetos, E. (2019). Audio-based identification of beehive states. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8256-8260). McCauley, R. (2016). Long-term monitoring of soundscapes and deciphering a usable index: Examples of fish choruses Sociobiology 67(4): 566-571 (December, 2020) 571 from Australia. In Proceedings of Meetings on Acoustics (p. 010023). Dublin-Ireland. Potts, S.G., Biesmeijer, J.C., Kremen, C., Neumann, P., Schweiger, O., Kunin, W.E. (2010). Global pollinator declines: trends, impacts and drivers. Trends in Ecology and Evolution. 25: 345-353. doi: 10.1016/j.tree.2010.01.007 Qandour, A., Ahmad, I., Habibi, D., & Leppard, M. ( 2014). Remote beehive monitoring using acoustic signals. Acoustic Australia, 42: 204-209. Robles-Guerrero, A., Saucedo-Anaya, T., González-Ramérez, E. & Galván-Tejada, C.E. (2017). Frequency analysis of honey bee buzz for automatic recognition of health status: A preliminary study. Research in Comput Science, 142: 89-98. doi: 10.13053/rcs-142-1-9 Rybin, V.G., Butusov, D.N., Karimov, T.I., Belkin, D.A. & Kozak, M.N. (2017). Embedded data acquisition system for beehive monitoring. In 2017 IEEE II International Conference on Control in Technical Systems (pp. 387-390). Seccomb, R.A. (2014). Autonomous reporting and tracking of pesticide incidents in honey bee colonies. https://www.sbir. gov/printpdf/708862. (accessed date: 21 September, 2020). Sharif, M.Z., Jiang, X. & Puswal, S.M. (2020). Pests, parasitoids, and predators: Can they degrade the sociality of a honeybee colony, and be assessed via acoustically monitored systems? Journal of Entomology and Zoology Studies, 8: 1248-1260. TeamRCore. (2015) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Villanueva-Rivera, L. & Pijanowski, B. (2016). Soundecology: soundscape ecology. R package version 1.3. 2. Yao, J., Zhu, Y.C. & Adamczyk, J. (2018). Responses of honey bees to lethal and sublethal doses of formulated clothianidin alone and mixtures. Journal of Economic Entomology, 111: 1517-1525. doi: 10.1093/jee/toy140. Yang, Y., Ma, S., Liu, F., Wang, Q., Wang, X., Hou, C., Wu, Y., Gao, J., Zhang, L., Liu, Y. & Diao, Q. (2020). Acute and chronic toxicity of acetamiprid, carbaryl, cypermethrin and deltamethrin to Apis mellifera larvae reared in vitro. Pest Management Science, 76: 978-985. doi: 10.1002/ps.5606. Zgank, A., 2020. Bee swarm activity acoustic classification for an IoT-based farm service. Sensors. 20: 21. doi: 10.33 90/ s20010021